[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:31 dist-test-slave-fb1v krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:31 dist-test-slave-fb1v krb5kdc[399](info): set up 2 sockets
Jun 20 08:54:31 dist-test-slave-fb1v krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 08:54:31.995559 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:32.114717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:32.220371 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:32.220619 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:32.220681 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:32.224674 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:32.224732 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:32.224759 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:32.224779 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:32.224800 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:32.229934 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:32.230906 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:32.231930 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:32.240015 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.239966 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.239989 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:32.240245 412 server_base.cc:1034] running on GCE node
I20260620 08:54:32.240733 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:32.241756 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:32.242941 412 hybrid_clock.cc:648] HybridClock initialized: now 1781945672242887 us; error 70 us; skew 500 ppm
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:32.245813 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:32.246683 412 webserver.cc:460] Webserver started at http://127.0.98.62:40883/ using document root <none> and password file <none>
I20260620 08:54:32.247186 412 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:32.247247 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:32.247430 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:32.249387 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "f17f981927694fd7ab74c275f4c8b21f"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "1543b8d89ed65780e121a45607127a65"
server_key_iv: "82f2545a8fc1d59f818f4017fbab90de"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.249969 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "f17f981927694fd7ab74c275f4c8b21f"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "1543b8d89ed65780e121a45607127a65"
server_key_iv: "82f2545a8fc1d59f818f4017fbab90de"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.253602 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:32.256260 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.257656 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:32.257777 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "f17f981927694fd7ab74c275f4c8b21f"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "1543b8d89ed65780e121a45607127a65"
server_key_iv: "82f2545a8fc1d59f818f4017fbab90de"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.257885 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:32.275254 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:32.278294 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:32.278473 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:32.284855 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44523
I20260620 08:54:32.284881 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44523 every 8 connection(s)
I20260620 08:54:32.285740 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:32.288434 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:32.290604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 412
I20260620 08:54:32.290741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:32.292016 392 external_mini_cluster.cc:1428] Setting key 3f6992f2b4fc7daacb0b8e7c2d38504f
I20260620 08:54:32.293833 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f: Bootstrap starting.
I20260620 08:54:32.296209 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:32.296912 480 log.cc:824] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:32.298612 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f: No bootstrap required, opened a new log
I20260620 08:54:32.301923 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17f981927694fd7ab74c275f4c8b21f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260620 08:54:32.302129 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:32.302182 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17f981927694fd7ab74c275f4c8b21f, State: Initialized, Role: FOLLOWER
I20260620 08:54:32.302608 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17f981927694fd7ab74c275f4c8b21f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260620 08:54:32.302743 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:32.302805 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:32.302898 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:32.303741 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17f981927694fd7ab74c275f4c8b21f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260620 08:54:32.304039 480 leader_election.cc:302] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f17f981927694fd7ab74c275f4c8b21f; no voters:
I20260620 08:54:32.304303 480 leader_election.cc:288] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:32.304387 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:32.304545 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [term 1 LEADER]: Becoming Leader. State: Replica: f17f981927694fd7ab74c275f4c8b21f, State: Running, Role: LEADER
I20260620 08:54:32.304885 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17f981927694fd7ab74c275f4c8b21f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260620 08:54:32.305510 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:32.306056 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f17f981927694fd7ab74c275f4c8b21f. Latest consensus state: current_term: 1 leader_uuid: "f17f981927694fd7ab74c275f4c8b21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17f981927694fd7ab74c275f4c8b21f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260620 08:54:32.306182 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:32.306452 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f17f981927694fd7ab74c275f4c8b21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f17f981927694fd7ab74c275f4c8b21f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260620 08:54:32.306531 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:32.306835 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:32.310379 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:32.316057 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:32.294866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.0.98.62:44523)
0620 08:54:32.295323 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:32.295341 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:32.296331 (+ 990us) server_negotiation.cc:407] Connection header received
0620 08:54:32.299305 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:32.299321 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:32.299667 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:32.299955 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:32.301965 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.303039 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.304242 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.304855 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.304998 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:32.309680 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:32.309687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:32.309703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:32.309733 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:32.312049 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:32.312707 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:32.312712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:32.312718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:32.312871 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:32.313245 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:32.313248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:32.313250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:32.313718 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:32.313895 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:32.314548 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 08:54:32.314833 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":128,"threads_started":1}
I20260620 08:54:32.316172 491 catalog_manager.cc:1347] Generated new cluster ID: b7a1158b40164f6e9723787d241e8a0c
I20260620 08:54:32.316430 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:32.328815 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:32.329695 491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:32.334869 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f17f981927694fd7ab74c275f4c8b21f: Generated new TSK 0
I20260620 08:54:32.335666 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:32.364712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:32.471194 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:32.471438 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:32.471498 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:32.475452 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:32.475507 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:32.475589 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:32.480747 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:32.481742 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:32.482751 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:32.491888 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.491918 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.491918 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:32.492025 506 server_base.cc:1034] running on GCE node
I20260620 08:54:32.492537 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:32.493130 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:32.494305 506 hybrid_clock.cc:648] HybridClock initialized: now 1781945672494283 us; error 34 us; skew 500 ppm
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:32.496978 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:32.497813 506 webserver.cc:460] Webserver started at http://127.0.98.1:37597/ using document root <none> and password file <none>
I20260620 08:54:32.498327 506 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:32.498394 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:32.498607 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:32.500619 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "4a59c5a838944ed1b7fdde347920ee85"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "fd9dd919fbf8a8cb427f422250b9b8a5"
server_key_iv: "e4302a787a6e4c07bb0753a07c75f544"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.501147 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4a59c5a838944ed1b7fdde347920ee85"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "fd9dd919fbf8a8cb427f422250b9b8a5"
server_key_iv: "e4302a787a6e4c07bb0753a07c75f544"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.504657 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:32.507306 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.508572 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:32.508685 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "4a59c5a838944ed1b7fdde347920ee85"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "fd9dd919fbf8a8cb427f422250b9b8a5"
server_key_iv: "e4302a787a6e4c07bb0753a07c75f544"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.508801 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:32.535722 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:32.538820 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:32.539000 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:32.539595 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:32.540460 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:32.540508 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.540561 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:32.540589 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.550417 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35907
I20260620 08:54:32.550452 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35907 every 8 connection(s)
I20260620 08:54:32.551288 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:32.560643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 506
I20260620 08:54:32.560758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:32.560998 392 external_mini_cluster.cc:1428] Setting key d7b7f333d1d282e1685568087a93928f
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:32.565709 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:32.553335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60791 (local address 127.0.98.62:44523)
0620 08:54:32.553492 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:32.553498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:32.554752 (+ 1254us) server_negotiation.cc:407] Connection header received
0620 08:54:32.556113 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:32.556117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:32.556172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:32.556245 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:32.557473 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.558004 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.559121 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.559351 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.559440 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:32.561960 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:32.561964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:32.561967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:32.562005 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:32.563724 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:32.564300 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:32.564303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:32.564305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:32.564384 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:32.564692 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:32.564695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:32.564697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:32.564871 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:32.564959 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:32.565418 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 08:54:32.565538 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:54:32.566557 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:32.553834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.1:60791)
0620 08:54:32.554590 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:32.554649 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:32.555900 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:32.556438 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:32.556445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:32.556818 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:32.557327 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:32.557340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.558117 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.558120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:32.558997 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:32.559006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.559484 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.559488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:32.559706 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:32.560257 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:32.560274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:32.561755 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:32.563840 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:32.563846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:32.563879 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:32.564193 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:32.564469 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:32.564472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:32.564474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:32.564608 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:32.564963 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:32.564969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:32.565204 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 08:54:32.565369 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 08:54:32.565675 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":130,"threads_started":1}
I20260620 08:54:32.568122 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260620 08:54:32.568396 635 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:32.568991 635 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:54:32.570605 444 ts_manager.cc:194] Registered new tserver with Master: 4a59c5a838944ed1b7fdde347920ee85 (127.0.98.1:35907)
I20260620 08:54:32.571934 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:32.586676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:32.692186 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:32.692417 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:32.692464 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:32.696359 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:32.696416 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:32.696481 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:32.701555 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:32.702550 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:32.703502 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:32.713038 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.713083 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:32.713089 640 server_base.cc:1034] running on GCE node
W20260620 08:54:32.713055 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:32.713632 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:32.714222 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:32.715377 640 hybrid_clock.cc:648] HybridClock initialized: now 1781945672715352 us; error 40 us; skew 500 ppm
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:32.717933 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:32.718724 640 webserver.cc:460] Webserver started at http://127.0.98.2:37179/ using document root <none> and password file <none>
I20260620 08:54:32.719226 640 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:32.719287 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:32.719460 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:32.721392 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "e32b6d638ac04b6d8e6fc07eafe0f943"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "3647d464a132269e3b79cde424894f4e"
server_key_iv: "40ccf2cb6e243d35c38449a45cce9df3"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.721909 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e32b6d638ac04b6d8e6fc07eafe0f943"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "3647d464a132269e3b79cde424894f4e"
server_key_iv: "40ccf2cb6e243d35c38449a45cce9df3"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.725559 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:32.727857 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.729051 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:54:32.729171 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "e32b6d638ac04b6d8e6fc07eafe0f943"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "3647d464a132269e3b79cde424894f4e"
server_key_iv: "40ccf2cb6e243d35c38449a45cce9df3"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.729261 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:32.763121 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:32.766335 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:32.766510 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:32.767089 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:32.767927 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:32.767975 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.768026 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:32.768052 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.778479 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40211
I20260620 08:54:32.778584 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40211 every 8 connection(s)
I20260620 08:54:32.779490 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:32.783079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 640
I20260620 08:54:32.783216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:32.783488 392 external_mini_cluster.cc:1428] Setting key 1c6dfe4e8b180cb41153e7ce0ea36564
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:54:32.794148 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:32.781317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33603 (local address 127.0.98.62:44523)
0620 08:54:32.781456 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:32.781460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:32.782368 (+ 908us) server_negotiation.cc:407] Connection header received
0620 08:54:32.784110 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:32.784114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:32.784171 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:32.784246 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:32.785616 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.786143 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.787315 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.787658 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.787755 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:32.790360 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:32.790364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:32.790367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:32.790395 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:32.791917 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:32.792569 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:32.792594 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:32.792595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:32.792692 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:32.793039 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:32.793042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:32.793044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:32.793270 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:32.793377 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:32.793865 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 08:54:32.793984 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:54:32.795207 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:32.781702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.2:33603)
0620 08:54:32.782183 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:32.782231 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:32.783896 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:32.784445 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:32.784453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:32.784805 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:32.785461 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:32.785480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.786265 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.786277 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:32.787196 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:32.787205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:32.787798 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:32.787802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:32.787951 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:32.788550 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:32.788572 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:32.790189 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:32.792042 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:32.792046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:32.792059 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:32.792324 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:32.792805 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:32.792808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:32.792810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:32.792947 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:32.793382 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:32.793389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:32.793636 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 08:54:32.793806 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 08:54:32.794219 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260620 08:54:32.796492 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260620 08:54:32.796767 769 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:32.797354 769 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260620 08:54:32.798441 444 ts_manager.cc:194] Registered new tserver with Master: e32b6d638ac04b6d8e6fc07eafe0f943 (127.0.98.2:40211)
I20260620 08:54:32.799029 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:32.811964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:32.922595 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:32.922857 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:32.922901 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:32.927095 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:32.927186 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:32.927268 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:32.933758 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:32.934902 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:32.935943 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:32.945614 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.945616 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:32.945721 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:32.945785 774 server_base.cc:1034] running on GCE node
I20260620 08:54:32.946303 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:32.946950 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:32.948127 774 hybrid_clock.cc:648] HybridClock initialized: now 1781945672948095 us; error 45 us; skew 500 ppm
Jun 20 08:54:32 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:32.950778 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:32.951601 774 webserver.cc:460] Webserver started at http://127.0.98.3:35103/ using document root <none> and password file <none>
I20260620 08:54:32.952144 774 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:32.952199 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:32.952365 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:32.954306 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "4953cdfbcb704e638da91941dfb092c7"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "1560f0ff226ffe680ff9f44f2422d04b"
server_key_iv: "70a45a192e502bba30ee9f6324d6ff20"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.954851 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4953cdfbcb704e638da91941dfb092c7"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "1560f0ff226ffe680ff9f44f2422d04b"
server_key_iv: "70a45a192e502bba30ee9f6324d6ff20"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.958475 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 08:54:32.960960 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.962190 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:32.962304 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "4953cdfbcb704e638da91941dfb092c7"
format_stamp: "Formatted at 2026-06-20 08:54:32 on dist-test-slave-fb1v"
server_key: "1560f0ff226ffe680ff9f44f2422d04b"
server_key_iv: "70a45a192e502bba30ee9f6324d6ff20"
server_key_version: "encryptionkey@0"
I20260620 08:54:32.962404 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:32.991050 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:32.994663 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:32.994825 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:32.995368 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:32.996174 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:32.996223 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:32.996263 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:32.996276 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:33.005903 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36327
I20260620 08:54:33.005915 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36327 every 8 connection(s)
I20260620 08:54:33.006868 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:33.008152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 774
I20260620 08:54:33.008240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:33.008487 392 external_mini_cluster.cc:1428] Setting key 3f4adad50845d44225d3de650e08fa61
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:33.021855 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:33.008715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58785 (local address 127.0.98.62:44523)
0620 08:54:33.008855 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:33.008859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:33.010367 (+ 1508us) server_negotiation.cc:407] Connection header received
0620 08:54:33.012268 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:33.012270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:33.012320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:33.012397 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:33.013879 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.014414 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.015908 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.016129 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.016249 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:33.018719 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:33.018722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:33.018724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:33.018748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:33.019990 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:33.020454 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:33.020458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:33.020460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:33.020536 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:33.020850 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:33.020853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:33.020855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:33.021026 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:33.021107 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:33.021579 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 08:54:33.021697 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:54:33.022781 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:33.009203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.3:58785)
0620 08:54:33.010116 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:33.010199 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:33.011829 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:33.012698 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:33.012708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:33.013147 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:33.013721 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:33.013745 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.014561 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.014564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:33.015779 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:33.015789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.016306 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.016310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:33.016480 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:33.017044 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:33.017061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:33.018547 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:33.020091 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:33.020096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:33.020113 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:33.020365 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:33.020624 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:33.020627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:33.020629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:33.020767 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:33.021118 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:33.021125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:33.021360 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 08:54:33.021530 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 08:54:33.021863 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":335,"threads_started":1}
I20260620 08:54:33.023814 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260620 08:54:33.024030 903 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:33.024469 903 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260620 08:54:33.025453 444 ts_manager.cc:194] Registered new tserver with Master: 4953cdfbcb704e638da91941dfb092c7 (127.0.98.3:36327)
I20260620 08:54:33.025993 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58785
I20260620 08:54:33.034464 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945673, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945673, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:33.058501 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:33.049397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.0.98.1:35907)
0620 08:54:33.049661 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:33.049673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:33.049696 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:54:33.049816 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:33.049822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:33.050002 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:33.050116 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:33.050689 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.051486 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.052376 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.052752 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.052835 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:33.055145 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:33.055154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:33.055168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:33.055193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:33.056614 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:33.056980 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:33.056985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:33.056986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:33.057065 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:33.057608 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:33.057611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:33.057612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:33.057846 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:33.058011 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:33.058200 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 08:54:33.058346 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
W20260620 08:54:33.059456 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49012
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:33.085155 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:33.076256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.0.98.1:35907)
0620 08:54:33.076412 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:33.076416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:33.076475 (+ 59us) server_negotiation.cc:407] Connection header received
0620 08:54:33.076588 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:33.076590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:33.076632 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:33.076707 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:33.077313 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.077853 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.078791 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:33.079025 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:33.079117 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:33.081600 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:33.081603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:33.081605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:33.081629 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:33.083069 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:33.083771 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:33.083774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:33.083775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:33.083859 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:33.084249 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:33.084252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:33.084254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:33.084468 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:33.084613 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:33.084846 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:54:33.084990 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:54:33.086378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 506
I20260620 08:54:33.091660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 640
I20260620 08:54:33.098919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 774
I20260620 08:54:33.104050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 412
2026-06-20T08:54:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2091 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[917](info): set up 2 sockets
Jun 20 08:54:33 dist-test-slave-fb1v krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 08:54:34.075019 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:34.191076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:34.298519 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:34.298796 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:34.298836 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:34.302845 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:34.302904 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:34.302922 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:34.302938 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:34.302953 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:34.308141 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:34.309098 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:34.310019 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:34.318786 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:34.318920 930 server_base.cc:1034] running on GCE node
W20260620 08:54:34.318833 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:34.318830 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:34.319664 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:34.320775 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:34.321981 930 hybrid_clock.cc:648] HybridClock initialized: now 1781945674321963 us; error 31 us; skew 500 ppm
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:34.325191 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:34.326102 930 webserver.cc:460] Webserver started at http://127.0.98.62:44353/ using document root <none> and password file <none>
I20260620 08:54:34.326661 930 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:34.326725 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:34.326889 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:34.328920 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "2e0af692376d4345a10756d67e737e98"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "4a729ebd7b64ce87fd98de8baac95ca8"
server_key_iv: "eb81e366ecc21b5d0a55200e5e343316"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.329429 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e0af692376d4345a10756d67e737e98"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "4a729ebd7b64ce87fd98de8baac95ca8"
server_key_iv: "eb81e366ecc21b5d0a55200e5e343316"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.333146 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:34.335894 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.337246 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:34.337356 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "2e0af692376d4345a10756d67e737e98"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "4a729ebd7b64ce87fd98de8baac95ca8"
server_key_iv: "eb81e366ecc21b5d0a55200e5e343316"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.337446 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:34.357463 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:34.360688 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:34.360852 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:34.367318 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34527
I20260620 08:54:34.367362 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34527 every 8 connection(s)
I20260620 08:54:34.368237 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:34.370988 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:34.376441 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98: Bootstrap starting.
I20260620 08:54:34.377236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 930
I20260620 08:54:34.377461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:34.377902 392 external_mini_cluster.cc:1428] Setting key 6058b497514ee4add7b2f4a180e37682
I20260620 08:54:34.379042 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:34.379792 998 log.cc:824] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:34.381771 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98: No bootstrap required, opened a new log
I20260620 08:54:34.385926 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e0af692376d4345a10756d67e737e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260620 08:54:34.386232 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:34.386307 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e0af692376d4345a10756d67e737e98, State: Initialized, Role: FOLLOWER
I20260620 08:54:34.386755 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e0af692376d4345a10756d67e737e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:34.386883 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:34.386948 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:34.387043 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:34.388537 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e0af692376d4345a10756d67e737e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260620 08:54:34.388928 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e0af692376d4345a10756d67e737e98; no voters:
I20260620 08:54:34.389240 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:34.389459 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:34.389772 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [term 1 LEADER]: Becoming Leader. State: Replica: 2e0af692376d4345a10756d67e737e98, State: Running, Role: LEADER
I20260620 08:54:34.390141 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e0af692376d4345a10756d67e737e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260620 08:54:34.390904 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:34.392053 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e0af692376d4345a10756d67e737e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e0af692376d4345a10756d67e737e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } } }
I20260620 08:54:34.392246 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:34.392693 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e0af692376d4345a10756d67e737e98. Latest consensus state: current_term: 1 leader_uuid: "2e0af692376d4345a10756d67e737e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e0af692376d4345a10756d67e737e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } } }
I20260620 08:54:34.392824 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:34.394361 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:34.379264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.98.62:34527)
0620 08:54:34.379714 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:34.379730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:34.379764 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:54:34.380406 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:34.380424 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:34.380755 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:34.381029 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:34.381843 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.382855 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.384033 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.384502 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.384639 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:34.387758 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:34.387767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:34.387790 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:34.387846 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:34.390305 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:34.391037 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:34.391044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:34.391049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:34.391238 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:34.391576 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:34.391579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:34.391581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:34.392152 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:34.392937 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:34.393074 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 08:54:34.393371 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
W20260620 08:54:34.396522 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:54:34.396587 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:54:34.396641 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:34.397399 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:34.402825 1012 catalog_manager.cc:1347] Generated new cluster ID: 2363f2b810b44014a6c593b5fc68bc9e
I20260620 08:54:34.402894 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:34.411633 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:34.412842 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:34.430661 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e0af692376d4345a10756d67e737e98: Generated new TSK 0
I20260620 08:54:34.431651 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:34.462706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:34.573100 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:34.573362 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:34.573410 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:34.577718 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:34.577788 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:34.577852 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:34.583246 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:34.584254 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:34.585158 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:34.594869 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:34.594900 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:34.594884 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:34.595069 1024 server_base.cc:1034] running on GCE node
I20260620 08:54:34.595633 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:34.596295 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:34.597510 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781945674597469 us; error 60 us; skew 500 ppm
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:34.600746 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:34.601608 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45445/ using document root <none> and password file <none>
I20260620 08:54:34.602150 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:34.602209 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:34.602404 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:34.604393 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ab0d5ce6dd64d889707ad92f19be814"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "d9f7d65c91c61a3604a138e133e98f78"
server_key_iv: "0d11124ffbca23381ef57af2684fc4c7"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.604904 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ab0d5ce6dd64d889707ad92f19be814"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "d9f7d65c91c61a3604a138e133e98f78"
server_key_iv: "0d11124ffbca23381ef57af2684fc4c7"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.608418 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:34.610733 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.612061 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:34.612164 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "5ab0d5ce6dd64d889707ad92f19be814"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "d9f7d65c91c61a3604a138e133e98f78"
server_key_iv: "0d11124ffbca23381ef57af2684fc4c7"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.612254 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:34.622852 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:34.625733 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:34.625892 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:34.626422 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:34.627209 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:34.627254 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.627303 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:34.627332 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.635563 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46451
I20260620 08:54:34.635597 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46451 every 8 connection(s)
I20260620 08:54:34.636405 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:34.638743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1024
I20260620 08:54:34.638842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:34.639094 392 external_mini_cluster.cc:1428] Setting key f3ddfc76bbec301c2e8b12cb19c3a552
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:54:34.652208 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:34.638188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54537 (local address 127.0.98.62:34527)
0620 08:54:34.638351 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:34.638355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:34.639188 (+ 833us) server_negotiation.cc:407] Connection header received
0620 08:54:34.641267 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:34.641272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:34.641356 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:34.641491 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:34.643161 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.643728 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.645295 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.645545 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.645621 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:34.648237 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:34.648242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:34.648245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:34.648271 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:34.650042 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:34.650612 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:34.650615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:34.650616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:34.650694 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:34.651030 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:34.651032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:34.651033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:34.651273 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:34.651347 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:34.651906 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 08:54:34.652041 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:54:34.653321 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:34.638545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34527 (local address 127.0.98.1:54537)
0620 08:54:34.639001 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:34.639048 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:34.640921 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:34.641760 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:34.641769 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:34.642265 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:34.642956 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:34.642978 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.643838 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.643841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:34.645151 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:34.645163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.645678 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.645681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:34.645850 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:34.646453 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:34.646469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:34.648069 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:34.650161 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:34.650167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:34.650180 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:34.650517 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:34.650805 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:34.650807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:34.650809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:34.650944 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:34.651369 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:34.651375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:34.651643 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 08:54:34.651834 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 08:54:34.652220 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260620 08:54:34.654584 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34527
I20260620 08:54:34.654837 1153 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:34.655372 1153 heartbeater.cc:511] Master 127.0.98.62:34527 requested a full tablet report, sending...
I20260620 08:54:34.657101 962 ts_manager.cc:194] Registered new tserver with Master: 5ab0d5ce6dd64d889707ad92f19be814 (127.0.98.1:46451)
I20260620 08:54:34.658612 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:34.668524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:34.783514 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:34.783789 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:34.783833 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:34.787997 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:34.788054 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:34.788117 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:34.793397 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:34.794341 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:34.795300 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:34.804364 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:34.804375 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:34.804374 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:34.804576 1158 server_base.cc:1034] running on GCE node
I20260620 08:54:34.805043 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:34.805673 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:34.806838 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781945674806816 us; error 34 us; skew 500 ppm
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:34.809638 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:34.810983 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46813/ using document root <none> and password file <none>
I20260620 08:54:34.811832 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:34.811918 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:34.812132 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:34.814219 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "43fc2aa1148243be9e1e126c5031d17a"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "35cbfcb04859b8a3dcb55103ebb8602a"
server_key_iv: "23e0675ba1176aff3505c3c00496e8d9"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.815056 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43fc2aa1148243be9e1e126c5031d17a"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "35cbfcb04859b8a3dcb55103ebb8602a"
server_key_iv: "23e0675ba1176aff3505c3c00496e8d9"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.819125 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 08:54:34.821557 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.822839 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 08:54:34.822955 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "43fc2aa1148243be9e1e126c5031d17a"
format_stamp: "Formatted at 2026-06-20 08:54:34 on dist-test-slave-fb1v"
server_key: "35cbfcb04859b8a3dcb55103ebb8602a"
server_key_iv: "23e0675ba1176aff3505c3c00496e8d9"
server_key_version: "encryptionkey@0"
I20260620 08:54:34.823058 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:34.838438 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:34.841535 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:34.841683 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:34.842295 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:34.843094 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:34.843137 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.843175 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:34.843204 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:34.851330 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38089
I20260620 08:54:34.851368 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38089 every 8 connection(s)
I20260620 08:54:34.852214 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:34.854326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1158
I20260620 08:54:34.854432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:34.854664 392 external_mini_cluster.cc:1428] Setting key 1fe1d69a62739289f69f7b29c1924a00
Jun 20 08:54:34 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:54:34.864897 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:34.853854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55547 (local address 127.0.98.62:34527)
0620 08:54:34.853988 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:34.853991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:34.854767 (+ 776us) server_negotiation.cc:407] Connection header received
0620 08:54:34.856264 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:34.856267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:34.856314 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:34.856384 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:34.857470 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.857984 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.859100 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.859309 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.859390 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:34.861661 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:34.861664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:34.861666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:34.861686 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:34.863060 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:34.863506 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:34.863509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:34.863511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:34.863580 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:34.863914 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:34.863917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:34.863919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:34.864080 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:34.864160 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:34.864621 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 08:54:34.864730 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:54:34.865617 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:34.854193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34527 (local address 127.0.98.2:55547)
0620 08:54:34.854629 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:34.854677 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:34.856075 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:34.856553 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:34.856560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:34.856884 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:34.857344 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:34.857357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.858091 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.858094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:34.858988 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:34.858998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:34.859434 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:34.859438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:34.859585 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:34.860174 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:34.860189 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:34.861513 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:34.863159 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:34.863164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:34.863178 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:34.863418 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:34.863693 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:34.863696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:34.863698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:34.863828 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:34.864173 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:34.864179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:34.864411 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 08:54:34.864568 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 08:54:34.864863 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260620 08:54:34.866567 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34527
I20260620 08:54:34.866803 1287 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:34.867242 1287 heartbeater.cc:511] Master 127.0.98.62:34527 requested a full tablet report, sending...
I20260620 08:54:34.868175 962 ts_manager.cc:194] Registered new tserver with Master: 43fc2aa1148243be9e1e126c5031d17a (127.0.98.2:38089)
I20260620 08:54:34.868677 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:54:35.661235 1153 heartbeater.cc:503] Master 127.0.98.62:34527 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260620 08:54:35.723203 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.867s user 0.000s sys 0.003s
I20260620 08:54:35.723778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:35.837253 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:35.837617 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:35.837699 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:35.843298 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:35.843379 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:35.843461 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:35.851240 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34527
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:35.852635 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:35.853993 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:35.865310 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:35.865304 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:35.865557 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:35.865865 1292 server_base.cc:1034] running on GCE node
I20260620 08:54:35.866668 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:35.867566 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:35.868829 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781945675868751 us; error 90 us; skew 500 ppm
I20260620 08:54:35.870648 1287 heartbeater.cc:503] Master 127.0.98.62:34527 was elected leader, sending a full tablet report...
Jun 20 08:54:35 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:35.872432 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:35.873404 1292 webserver.cc:460] Webserver started at http://127.0.98.3:40109/ using document root <none> and password file <none>
I20260620 08:54:35.873980 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:35.874276 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:35.874506 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:35.878110 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8c92dcef340409e9664ceb3949f33c4"
format_stamp: "Formatted at 2026-06-20 08:54:35 on dist-test-slave-fb1v"
server_key: "2c93ad6cbd1013533172726ba3f282a6"
server_key_iv: "d782df3e82e7aba71e038ef38a5b2222"
server_key_version: "encryptionkey@0"
I20260620 08:54:35.879201 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8c92dcef340409e9664ceb3949f33c4"
format_stamp: "Formatted at 2026-06-20 08:54:35 on dist-test-slave-fb1v"
server_key: "2c93ad6cbd1013533172726ba3f282a6"
server_key_iv: "d782df3e82e7aba71e038ef38a5b2222"
server_key_version: "encryptionkey@0"
I20260620 08:54:35.884616 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 08:54:35.887275 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:35.888800 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:35.888911 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "d8c92dcef340409e9664ceb3949f33c4"
format_stamp: "Formatted at 2026-06-20 08:54:35 on dist-test-slave-fb1v"
server_key: "2c93ad6cbd1013533172726ba3f282a6"
server_key_iv: "d782df3e82e7aba71e038ef38a5b2222"
server_key_version: "encryptionkey@0"
I20260620 08:54:35.889019 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:35.921192 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:35.924424 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:35.924602 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:35.925177 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:35.926079 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:35.926127 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:35.926209 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:35.926247 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:35.934923 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44933
I20260620 08:54:35.934994 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44933 every 8 connection(s)
I20260620 08:54:35.935858 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:35.940637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1292
I20260620 08:54:35.940766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:35.941015 392 external_mini_cluster.cc:1428] Setting key 06b98746973a39791b58584189d8a88c
Jun 20 08:54:35 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:35.950841 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.937661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57295 (local address 127.0.98.62:34527)
0620 08:54:35.937943 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:35.937947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:35.938624 (+ 677us) server_negotiation.cc:407] Connection header received
0620 08:54:35.940088 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:35.940091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:35.940158 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:35.940238 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:35.941567 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.942266 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.943494 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.943754 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.943844 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.946483 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:35.946486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:35.946489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:35.946515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:35.947980 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:35.948837 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:35.948843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:35.948846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:35.948970 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:35.949462 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:35.949466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:35.949467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:35.949716 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:35.949827 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:35.950573 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 08:54:35.950699 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260620 08:54:35.951706 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.938030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34527 (local address 127.0.98.3:57295)
0620 08:54:35.938462 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:35.938502 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:35.939881 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:35.940427 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:35.940434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:35.940815 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:35.941356 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.941371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.942401 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.942405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.943379 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.943388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.943900 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.943903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.944049 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.944701 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:35.944718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:35.946136 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:35.948188 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:35.948196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:35.948216 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:35.948639 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:35.949128 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:35.949131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:35.949133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:35.949315 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:35.949869 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:35.949885 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:35.950230 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 08:54:35.950426 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 08:54:35.950713 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260620 08:54:35.953040 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34527
I20260620 08:54:35.953279 1421 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:35.953810 1421 heartbeater.cc:511] Master 127.0.98.62:34527 requested a full tablet report, sending...
I20260620 08:54:35.954743 962 ts_manager.cc:194] Registered new tserver with Master: d8c92dcef340409e9664ceb3949f33c4 (127.0.98.3:44933)
I20260620 08:54:35.955307 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57295
I20260620 08:54:35.966868 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:54:35.977777 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.970339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.0.98.62:34527)
0620 08:54:35.970498 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:35.970502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:35.970581 (+ 79us) server_negotiation.cc:407] Connection header received
0620 08:54:35.970708 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:35.970736 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:35.970780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:35.970847 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:35.971531 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.972098 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.972886 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.973102 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.973185 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.974817 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:35.974820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:35.974822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:35.974851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:35.976215 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:35.976622 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:35.976626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:35.976628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:35.976704 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:35.976952 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:35.976955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:35.976957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:35.977171 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:35.977298 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:35.977510 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 08:54:35.977627 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 08:54:35.982595 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:54:35.985572 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:36.000082 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.994326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.0.98.1:46451)
0620 08:54:35.994673 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:35.994681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:35.995138 (+ 457us) server_negotiation.cc:407] Connection header received
0620 08:54:35.995215 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:35.995221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:35.995427 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:35.995554 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:35.995836 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.996943 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.998498 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.999396 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.999490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.999667 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:35.999845 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 08:54:35.999940 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":74,"threads_started":1}
I20260620 08:54:36.000077 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.994208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46451 (local address 127.0.0.1:53954)
0620 08:54:35.994825 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:35.994871 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:35.995003 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:35.995554 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:35.995559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:35.995578 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:35.995731 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.995738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.997051 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.997055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.998370 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.998379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.999501 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.999504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.999642 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.999705 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 08:54:35.999807 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:54:35.999925 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260620 08:54:36.000088 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.994926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38089 (local address 127.0.0.1:53846)
0620 08:54:35.995795 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:35.995805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:35.995881 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:35.996307 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:35.996310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:35.996327 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:35.996437 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.996443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.997548 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.997551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.998527 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.998535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.999650 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.999653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.999777 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.999801 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:54:35.999873 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:54:35.999941 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":25,"thread_start_us":80,"threads_started":1}
I20260620 08:54:36.000705 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.995297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.0.98.2:38089)
0620 08:54:35.995661 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:35.995668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:35.995879 (+ 211us) server_negotiation.cc:407] Connection header received
0620 08:54:35.995990 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:35.995995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:35.996171 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:35.996321 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:35.996543 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.997444 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.998637 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.999217 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.999330 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:35.999556 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.000305 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.000486 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":81,"threads_started":1}
I20260620 08:54:36.001070 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.994467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44933 (local address 127.0.0.1:48354)
0620 08:54:35.995324 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:35.995336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:35.995414 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:35.996554 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:35.996557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:35.996572 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:35.996627 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.996631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.997987 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.997990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:35.999085 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:35.999094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.000715 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.000718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:36.000814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.000837 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:54:36.000895 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:54:36.000957 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":99,"threads_started":1}
I20260620 08:54:36.001402 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:35.994892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.98.3:44933)
0620 08:54:35.996057 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:35.996064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:35.996080 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:54:35.996135 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:35.996138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:35.996279 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:35.996415 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:35.996733 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:35.997842 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:35.999268 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.000617 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.000792 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.001028 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.001078 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.001171 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":149,"threads_started":1}
I20260620 08:54:36.003628 1221 tablet_service.cc:1467] Processing CreateTablet for tablet ca106077bc324969955ecf0db37dd5e2 (DEFAULT_TABLE table=test-table [id=357bd240aef743d59b9114e8b8fa4d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:36.003705 1087 tablet_service.cc:1467] Processing CreateTablet for tablet ca106077bc324969955ecf0db37dd5e2 (DEFAULT_TABLE table=test-table [id=357bd240aef743d59b9114e8b8fa4d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:36.003713 1355 tablet_service.cc:1467] Processing CreateTablet for tablet ca106077bc324969955ecf0db37dd5e2 (DEFAULT_TABLE table=test-table [id=357bd240aef743d59b9114e8b8fa4d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:36.004669 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca106077bc324969955ecf0db37dd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:36.004788 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca106077bc324969955ecf0db37dd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:36.004788 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca106077bc324969955ecf0db37dd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:36.010789 1440 tablet_bootstrap.cc:492] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: Bootstrap starting.
I20260620 08:54:36.012323 1442 tablet_bootstrap.cc:492] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: Bootstrap starting.
I20260620 08:54:36.013733 1440 tablet_bootstrap.cc:654] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:36.014393 1442 tablet_bootstrap.cc:654] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:36.014569 1440 log.cc:824] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:36.015194 1442 log.cc:824] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:36.015262 1441 tablet_bootstrap.cc:492] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: Bootstrap starting.
I20260620 08:54:36.017252 1440 tablet_bootstrap.cc:492] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: No bootstrap required, opened a new log
I20260620 08:54:36.017292 1441 tablet_bootstrap.cc:654] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:36.017303 1442 tablet_bootstrap.cc:492] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: No bootstrap required, opened a new log
I20260620 08:54:36.017450 1440 ts_tablet_manager.cc:1397] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 08:54:36.017463 1442 ts_tablet_manager.cc:1397] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:54:36.018504 1441 log.cc:824] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:36.020617 1441 tablet_bootstrap.cc:492] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: No bootstrap required, opened a new log
I20260620 08:54:36.020797 1441 ts_tablet_manager.cc:1397] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 08:54:36.021921 1442 raft_consensus.cc:348] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.022164 1442 raft_consensus.cc:374] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:36.022209 1442 raft_consensus.cc:729] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab0d5ce6dd64d889707ad92f19be814, State: Initialized, Role: FOLLOWER
I20260620 08:54:36.023113 1442 consensus_queue.cc:260] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.023223 1440 raft_consensus.cc:348] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.023468 1440 raft_consensus.cc:374] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:36.023520 1440 raft_consensus.cc:729] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fc2aa1148243be9e1e126c5031d17a, State: Initialized, Role: FOLLOWER
I20260620 08:54:36.024051 1440 consensus_queue.cc:260] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.024235 1442 ts_tablet_manager.cc:1428] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260620 08:54:36.025058 1440 ts_tablet_manager.cc:1428] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 08:54:36.024523 1441 raft_consensus.cc:348] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.025655 1441 raft_consensus.cc:374] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:36.027395 1441 raft_consensus.cc:729] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c92dcef340409e9664ceb3949f33c4, State: Initialized, Role: FOLLOWER
I20260620 08:54:36.028247 1441 consensus_queue.cc:260] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.028893 1421 heartbeater.cc:503] Master 127.0.98.62:34527 was elected leader, sending a full tablet report...
I20260620 08:54:36.029235 1441 ts_tablet_manager.cc:1428] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260620 08:54:36.104153 1288 tablet.cc:2369] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:54:36.138283 1154 tablet.cc:2369] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:36.184887 1447 raft_consensus.cc:482] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:36.185115 1447 raft_consensus.cc:504] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.186333 1447 leader_election.cc:288] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ab0d5ce6dd64d889707ad92f19be814 (127.0.98.1:46451), d8c92dcef340409e9664ceb3949f33c4 (127.0.98.3:44933)
W20260620 08:54:36.186830 1422 tablet.cc:2369] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:36.190711 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.186947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42085 (local address 127.0.98.1:46451)
0620 08:54:36.187062 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:36.187067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:36.187083 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:54:36.187203 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:36.187207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:36.187261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:36.187307 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:36.187995 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.188739 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.189825 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.190215 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.190312 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.190387 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.190524 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.190594 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:54:36.190753 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.186582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44933 (local address 127.0.98.2:35403)
0620 08:54:36.187190 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:36.187202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:36.187327 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:36.187684 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:36.187688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:36.187707 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:36.187824 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:36.187831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.188829 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.188831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:36.189722 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:36.189731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.190336 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.190339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:36.190440 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.190460 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:54:36.190522 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 08:54:36.190592 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":23,"thread_start_us":129,"threads_started":1}
I20260620 08:54:36.190744 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.186691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35403 (local address 127.0.98.3:44933)
0620 08:54:36.186810 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:36.186814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:36.187260 (+ 446us) server_negotiation.cc:407] Connection header received
0620 08:54:36.187456 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:36.187459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:36.187506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:36.187577 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:36.187958 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.188700 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.189836 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.190230 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.190323 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.190385 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.190545 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.190626 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:54:36.190811 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.186574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46451 (local address 127.0.98.2:42085)
0620 08:54:36.186954 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:36.186968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:36.187068 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:36.187441 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:36.187444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:36.187461 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:36.187578 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:36.187584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.188841 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.188845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:36.189711 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:36.189719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.190317 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.190321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:36.190411 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.190439 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 08:54:36.190502 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:54:36.190571 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":49,"thread_start_us":134,"threads_started":1}
I20260620 08:54:36.191288 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca106077bc324969955ecf0db37dd5e2" candidate_uuid: "43fc2aa1148243be9e1e126c5031d17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c92dcef340409e9664ceb3949f33c4" is_pre_election: true
I20260620 08:54:36.191401 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca106077bc324969955ecf0db37dd5e2" candidate_uuid: "43fc2aa1148243be9e1e126c5031d17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab0d5ce6dd64d889707ad92f19be814" is_pre_election: true
I20260620 08:54:36.191555 1375 raft_consensus.cc:2454] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43fc2aa1148243be9e1e126c5031d17a in term 0.
I20260620 08:54:36.191704 1107 raft_consensus.cc:2454] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43fc2aa1148243be9e1e126c5031d17a in term 0.
I20260620 08:54:36.192032 1176 leader_election.cc:302] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43fc2aa1148243be9e1e126c5031d17a, d8c92dcef340409e9664ceb3949f33c4; no voters:
I20260620 08:54:36.192294 1447 raft_consensus.cc:2781] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:36.192377 1447 raft_consensus.cc:482] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:36.192420 1447 raft_consensus.cc:3037] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:36.219856 1447 raft_consensus.cc:504] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.220314 1447 leader_election.cc:288] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [CANDIDATE]: Term 1 election: Requested vote from peers 5ab0d5ce6dd64d889707ad92f19be814 (127.0.98.1:46451), d8c92dcef340409e9664ceb3949f33c4 (127.0.98.3:44933)
I20260620 08:54:36.220813 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca106077bc324969955ecf0db37dd5e2" candidate_uuid: "43fc2aa1148243be9e1e126c5031d17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab0d5ce6dd64d889707ad92f19be814"
I20260620 08:54:36.220826 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca106077bc324969955ecf0db37dd5e2" candidate_uuid: "43fc2aa1148243be9e1e126c5031d17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c92dcef340409e9664ceb3949f33c4"
I20260620 08:54:36.220948 1107 raft_consensus.cc:3037] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:36.220983 1375 raft_consensus.cc:3037] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:36.222227 1107 raft_consensus.cc:2454] T ca106077bc324969955ecf0db37dd5e2 P 5ab0d5ce6dd64d889707ad92f19be814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43fc2aa1148243be9e1e126c5031d17a in term 1.
I20260620 08:54:36.222244 1375 raft_consensus.cc:2454] T ca106077bc324969955ecf0db37dd5e2 P d8c92dcef340409e9664ceb3949f33c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43fc2aa1148243be9e1e126c5031d17a in term 1.
I20260620 08:54:36.222618 1175 leader_election.cc:302] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43fc2aa1148243be9e1e126c5031d17a, 5ab0d5ce6dd64d889707ad92f19be814; no voters:
I20260620 08:54:36.222812 1447 raft_consensus.cc:2781] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:36.223024 1447 raft_consensus.cc:686] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [term 1 LEADER]: Becoming Leader. State: Replica: 43fc2aa1148243be9e1e126c5031d17a, State: Running, Role: LEADER
I20260620 08:54:36.223456 1447 consensus_queue.cc:237] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } }
I20260620 08:54:36.226609 961 catalog_manager.cc:5495] T ca106077bc324969955ecf0db37dd5e2 P 43fc2aa1148243be9e1e126c5031d17a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43fc2aa1148243be9e1e126c5031d17a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "43fc2aa1148243be9e1e126c5031d17a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab0d5ce6dd64d889707ad92f19be814" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8c92dcef340409e9664ceb3949f33c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43fc2aa1148243be9e1e126c5031d17a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38089 } health_report { overall_health: HEALTHY } } }
Jun 20 08:54:36 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:36.246141 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.234684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.0.98.1:46451)
0620 08:54:36.234807 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:36.234811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:36.234921 (+ 110us) server_negotiation.cc:407] Connection header received
0620 08:54:36.235056 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:36.235059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:36.235104 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:36.235176 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:36.235838 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.236354 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.237188 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.237411 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.237490 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.242881 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:36.242892 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:36.242908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:36.242933 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:36.244470 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:36.244895 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:36.244899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:36.244901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:36.244971 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:36.245288 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:36.245291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:36.245292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:36.245559 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:36.245755 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.245919 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.246018 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 08:54:36 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:54:36 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:36.333479 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.323221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.0.98.1:46451)
0620 08:54:36.323534 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:36.323540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:36.323575 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:54:36.323847 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:36.323851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:36.323930 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:36.324081 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:36.325058 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.325657 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.326727 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.326969 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.327057 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.329822 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:36.329825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:36.329827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:36.329853 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:36.331338 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:36.331746 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:36.331750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:36.331751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:36.331826 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:36.332359 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:36.332363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:36.332364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:36.332685 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:36.332854 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.333227 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.333354 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 08:54:36 dist-test-slave-fb1v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:54:36 dist-test-slave-fb1v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:36.431067 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:36.421604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.1:46451)
0620 08:54:36.421783 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:36.421787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:36.421991 (+ 204us) server_negotiation.cc:407] Connection header received
0620 08:54:36.422117 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:36.422120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:36.422166 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:36.422243 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:36.423122 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.423663 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.424537 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:36.424802 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:36.424889 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:36.427770 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:36.427774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:36.427776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:36.427804 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:36.429127 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:36.429588 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:36.429591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:36.429592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:36.429673 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:36.429917 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:36.429919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:36.429920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:36.430085 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:36.430183 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:36.430574 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 08:54:36.430785 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 08:54:36.433109 1067 tablet_service.cc:2908] Rejecting scan request for tablet ca106077bc324969955ecf0db37dd5e2: Uninitialized: clean time has not yet been initialized
I20260620 08:54:36.435135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1024
I20260620 08:54:36.440186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1158
I20260620 08:54:36.445289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1292
I20260620 08:54:36.450019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 930
2026-06-20T08:54:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3347 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:37 dist-test-slave-fb1v krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:37 dist-test-slave-fb1v krb5kdc[1465](info): set up 2 sockets
Jun 20 08:54:37 dist-test-slave-fb1v krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260620 08:54:38.927053 392 mini_kdc.cc:121] Time spent starting KDC: real 2.455s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 08:54:39.089972 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.163s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 08:54:39.262374 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.172s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 08:54:39.490031 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.228s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/test-user.keytab.
W20260620 08:54:40.400812 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.911s user 0.000s sys 0.001s
Jun 20 08:54:40 dist-test-slave-fb1v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:40.442839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41393
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:40.552299 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:40.552544 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:40.552582 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:40.556473 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:40.556529 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:54:40.556545 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:40.556560 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:40.556576 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:40.556589 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:40.561990 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:40.562965 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:40.563915 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:40.572293 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:40.572255 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:40.572304 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:40.572422 1478 server_base.cc:1034] running on GCE node
I20260620 08:54:40.573119 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:40.574119 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:40.575273 1478 hybrid_clock.cc:648] HybridClock initialized: now 1781945680575251 us; error 30 us; skew 500 ppm
Jun 20 08:54:40 dist-test-slave-fb1v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:40.578280 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:40.579409 1478 webserver.cc:460] Webserver started at http://127.0.98.62:42003/ using document root <none> and password file <none>
I20260620 08:54:40.580127 1478 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:40.580214 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:40.580454 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:40.582661 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "9041f14be1954768b58e2921603578e9"
format_stamp: "Formatted at 2026-06-20 08:54:40 on dist-test-slave-fb1v"
server_key: "6f907661da1aa90fa458b6ac3e4ac4e3"
server_key_iv: "6e01c63ec59a655d6fb8fac97052ff3e"
server_key_version: "encryptionkey@0"
I20260620 08:54:40.583217 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "9041f14be1954768b58e2921603578e9"
format_stamp: "Formatted at 2026-06-20 08:54:40 on dist-test-slave-fb1v"
server_key: "6f907661da1aa90fa458b6ac3e4ac4e3"
server_key_iv: "6e01c63ec59a655d6fb8fac97052ff3e"
server_key_version: "encryptionkey@0"
I20260620 08:54:40.586959 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:54:40.589361 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:40.590548 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 08:54:40.590658 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "9041f14be1954768b58e2921603578e9"
format_stamp: "Formatted at 2026-06-20 08:54:40 on dist-test-slave-fb1v"
server_key: "6f907661da1aa90fa458b6ac3e4ac4e3"
server_key_iv: "6e01c63ec59a655d6fb8fac97052ff3e"
server_key_version: "encryptionkey@0"
I20260620 08:54:40.590747 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:40.620711 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:40.623674 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:40.623865 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:40.631178 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41393
I20260620 08:54:40.631276 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41393 every 8 connection(s)
I20260620 08:54:40.632115 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:40.634871 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:40.638718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1478
I20260620 08:54:40.638808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:40.639034 392 external_mini_cluster.cc:1428] Setting key 45ba5c4bf03083258e729c861460eec9
I20260620 08:54:40.640421 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: Bootstrap starting.
I20260620 08:54:40.643000 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:40.643762 1556 log.cc:824] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:40.645440 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: No bootstrap required, opened a new log
Jun 20 08:54:40 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:40.648734 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9041f14be1954768b58e2921603578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260620 08:54:40.648957 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:40.649011 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9041f14be1954768b58e2921603578e9, State: Initialized, Role: FOLLOWER
I20260620 08:54:40.649420 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9041f14be1954768b58e2921603578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260620 08:54:40.649544 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:40.649606 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:40.649699 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:40.650678 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9041f14be1954768b58e2921603578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260620 08:54:40.650962 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9041f14be1954768b58e2921603578e9; no voters:
I20260620 08:54:40.651262 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:40.651391 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:40.651678 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [term 1 LEADER]: Becoming Leader. State: Replica: 9041f14be1954768b58e2921603578e9, State: Running, Role: LEADER
I20260620 08:54:40.652055 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9041f14be1954768b58e2921603578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260620 08:54:40.652515 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:40.652845 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:40.640437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.0.98.62:41393)
0620 08:54:40.640863 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:40.640881 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:40.640920 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:54:40.641746 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:40.641768 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:40.642158 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:40.642521 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:40.643154 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.644204 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.645000 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.645456 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.645592 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:40.647762 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:40.647767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:40.647779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:40.647803 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:40.649629 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:40.650097 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:40.650102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:40.650106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:40.650209 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:40.650494 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:40.650505 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:40.650506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:40.650978 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:40.651168 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:40.651471 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 08:54:40.651820 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":144,"threads_started":1}
I20260620 08:54:40.653638 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9041f14be1954768b58e2921603578e9. Latest consensus state: current_term: 1 leader_uuid: "9041f14be1954768b58e2921603578e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9041f14be1954768b58e2921603578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } } }
I20260620 08:54:40.653772 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:40.654136 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:40.654452 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9041f14be1954768b58e2921603578e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9041f14be1954768b58e2921603578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } } }
I20260620 08:54:40.654541 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:40.657272 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:40.662923 1570 catalog_manager.cc:1347] Generated new cluster ID: 1fd9b8fcc6b74f1d9f7bdfaafcfb6562
I20260620 08:54:40.662982 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:40.692224 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:40.693178 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:40.697937 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: Generated new TSK 0
I20260620 08:54:40.698661 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:40.734948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:40.858139 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:40.858384 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:40.858426 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:40.862386 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:40.862442 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:40.862479 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:54:40.862511 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:40.867718 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:40.868727 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:40.869635 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:40.879388 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:40.879462 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:40.879477 1582 server_base.cc:1034] running on GCE node
W20260620 08:54:40.879400 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:40.880082 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:40.880676 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:40.881858 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781945680881821 us; error 28 us; skew 500 ppm
Jun 20 08:54:40 dist-test-slave-fb1v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:40.884416 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:40.885198 1582 webserver.cc:460] Webserver started at http://127.0.98.1:44687/ using document root <none> and password file <none>
I20260620 08:54:40.885691 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:40.885753 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:40.885936 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:40.887907 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "e734f57eac6949d88ac64a0753d248c1"
format_stamp: "Formatted at 2026-06-20 08:54:40 on dist-test-slave-fb1v"
server_key: "b98d5fae129898421a57fc54e2dc350e"
server_key_iv: "4fbb617668e3ab5fd3c1756c2cddf258"
server_key_version: "encryptionkey@0"
I20260620 08:54:40.888428 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e734f57eac6949d88ac64a0753d248c1"
format_stamp: "Formatted at 2026-06-20 08:54:40 on dist-test-slave-fb1v"
server_key: "b98d5fae129898421a57fc54e2dc350e"
server_key_iv: "4fbb617668e3ab5fd3c1756c2cddf258"
server_key_version: "encryptionkey@0"
I20260620 08:54:40.891912 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:54:40.894184 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:40.895370 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:40.895480 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "e734f57eac6949d88ac64a0753d248c1"
format_stamp: "Formatted at 2026-06-20 08:54:40 on dist-test-slave-fb1v"
server_key: "b98d5fae129898421a57fc54e2dc350e"
server_key_iv: "4fbb617668e3ab5fd3c1756c2cddf258"
server_key_version: "encryptionkey@0"
I20260620 08:54:40.895565 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:40.909587 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:40.912606 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:40.912787 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:40.914928 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:40.914984 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:40.915033 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:40.915052 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:54:40 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:40.923939 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33033
I20260620 08:54:40.924104 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33033 every 8 connection(s)
I20260620 08:54:40.924913 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:40.927983 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:40.915772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40987 (local address 127.0.98.62:41393)
0620 08:54:40.915893 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:40.915898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:40.916253 (+ 355us) server_negotiation.cc:407] Connection header received
0620 08:54:40.917205 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:40.917208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:40.917274 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:40.917347 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:40.918815 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.919694 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.921576 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.921818 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.921910 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:40.924605 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:40.924609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:40.924612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:40.924640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:40.926091 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:40.926560 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:40.926563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:40.926564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:40.926640 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:40.926970 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:40.926973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:40.926974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:40.927161 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:40.927246 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:40.927745 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 08:54:40.927853 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:54:40.928682 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:40.915543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.1:40987)
0620 08:54:40.916116 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:40.916155 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:40.916976 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:40.917531 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:40.917539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:40.917970 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:40.918611 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:40.918631 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.919852 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.919856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:40.921451 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:40.921461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.921956 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.921960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:40.922158 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:40.922835 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:40.922857 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:40.924426 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:40.926204 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:40.926208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:40.926219 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:40.926465 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:40.926708 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:40.926710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:40.926712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:40.926856 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:40.927254 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:40.927259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:40.927471 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 08:54:40.927656 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 08:54:40.927932 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":127,"threads_started":1}
I20260620 08:54:40.930034 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41393
I20260620 08:54:40.930159 1713 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:40.930631 1713 heartbeater.cc:511] Master 127.0.98.62:41393 requested a full tablet report, sending...
I20260620 08:54:40.931439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1582
I20260620 08:54:40.931547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:40.931849 392 external_mini_cluster.cc:1428] Setting key 93a7758438b2b268307dd67ec8f61f24
I20260620 08:54:40.932255 1509 ts_manager.cc:194] Registered new tserver with Master: e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033)
I20260620 08:54:40.933458 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40987
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:54:40.940738 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:40.932114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.1:50123)
0620 08:54:40.932383 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:40.932397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:40.932480 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:40.934364 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:40.934367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:40.934567 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:40.934680 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:40.934687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.935649 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.935653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:40.936222 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:40.936230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.936635 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.936639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:40.936723 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:40.937352 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:40.937363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:40.937689 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:40.939188 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:40.939191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:40.939193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:40.939369 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:40.939652 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:40.939656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:40.939658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:40.939767 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:40.940212 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:40.940215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:40.940301 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 08:54:40.940397 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 08:54:40.940528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
I20260620 08:54:40.940837 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:40.933986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50123 (local address 127.0.98.62:41393)
0620 08:54:40.934086 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:40.934090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:40.934102 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:54:40.934134 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:40.934137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:40.934177 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:40.934261 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:40.934800 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.935489 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.936329 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:40.936542 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:40.936625 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:40.937817 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:40.937821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:40.937824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:40.937847 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:40.939087 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:40.939451 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:40.939455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:40.939456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:40.939525 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:40.939896 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:40.939899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:40.939901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:40.940116 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:40.940226 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:40.940578 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 08:54:40.940699 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:40.956750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:41.066694 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:41.066932 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:41.066973 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:41.070866 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:41.070919 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:41.070956 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:54:41.070995 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:41.076246 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:41.077159 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:41.078071 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:41.087853 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:41.087893 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:41.087853 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:41.088089 1722 server_base.cc:1034] running on GCE node
I20260620 08:54:41.088616 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:41.089214 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:41.090372 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781945681090348 us; error 44 us; skew 500 ppm
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:41.093026 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:41.093895 1722 webserver.cc:460] Webserver started at http://127.0.98.2:32963/ using document root <none> and password file <none>
I20260620 08:54:41.094403 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:41.094465 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:41.094638 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:41.096593 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "54b995c465d34fa58cd9d703d969ee79"
format_stamp: "Formatted at 2026-06-20 08:54:41 on dist-test-slave-fb1v"
server_key: "348af890c3a11b32d50a36dec498bc72"
server_key_iv: "747cadfc5fdb71899bf06de016d50fd6"
server_key_version: "encryptionkey@0"
I20260620 08:54:41.097091 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54b995c465d34fa58cd9d703d969ee79"
format_stamp: "Formatted at 2026-06-20 08:54:41 on dist-test-slave-fb1v"
server_key: "348af890c3a11b32d50a36dec498bc72"
server_key_iv: "747cadfc5fdb71899bf06de016d50fd6"
server_key_version: "encryptionkey@0"
I20260620 08:54:41.100768 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:41.103132 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:41.104373 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:41.104480 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "54b995c465d34fa58cd9d703d969ee79"
format_stamp: "Formatted at 2026-06-20 08:54:41 on dist-test-slave-fb1v"
server_key: "348af890c3a11b32d50a36dec498bc72"
server_key_iv: "747cadfc5fdb71899bf06de016d50fd6"
server_key_version: "encryptionkey@0"
I20260620 08:54:41.104569 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:41.117082 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:41.119726 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:41.119886 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:41.121207 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:41.121256 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:41.121306 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:41.121333 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:41.131042 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37091
I20260620 08:54:41.131323 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37091 every 8 connection(s)
I20260620 08:54:41.132061 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:41.132385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1722
I20260620 08:54:41.132491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:41.132788 392 external_mini_cluster.cc:1428] Setting key 1ea0d2bae98b3118ff201cf4eeb29658
I20260620 08:54:41.134006 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.121848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33721 (local address 127.0.98.62:41393)
0620 08:54:41.121967 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.121972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.122936 (+ 964us) server_negotiation.cc:407] Connection header received
0620 08:54:41.123929 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.123933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.123991 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.124046 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.125423 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.126206 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.127345 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.127698 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.127810 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.130180 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.130184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.130186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.130211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.131849 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.132403 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.132407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.132409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.132500 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.132863 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.132866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.132868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.133073 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.133164 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.133747 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.133861 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:54:41.134774 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.122338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.2:33721)
0620 08:54:41.122797 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.122835 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.123705 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.124408 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.124417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.124764 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:41.125243 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.125261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.126338 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.126341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.127222 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.127231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.127852 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.127857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.128038 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.128554 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:41.128570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:41.130029 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:41.131948 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.131953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.131964 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.132284 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.132608 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.132611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.132614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.132767 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.133211 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:41.133217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:41.133457 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.133652 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.133985 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260620 08:54:41.136126 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41393
I20260620 08:54:41.136286 1853 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:41.136857 1853 heartbeater.cc:511] Master 127.0.98.62:41393 requested a full tablet report, sending...
I20260620 08:54:41.137862 1510 ts_manager.cc:194] Registered new tserver with Master: 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091)
I20260620 08:54:41.138536 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33721
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:54:41.146162 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.138167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56397 (local address 127.0.98.62:41393)
0620 08:54:41.138271 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.138275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.139884 (+ 1609us) server_negotiation.cc:407] Connection header received
0620 08:54:41.139968 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.139971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.140012 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.140074 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.140481 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.140973 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.141764 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.141971 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.142052 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.143201 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.143205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.143208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.143231 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.144627 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.145059 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.145062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.145064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.145137 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.145378 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.145381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.145383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.145577 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.145662 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.145957 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.146054 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:54:41.146209 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.138102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.2:56397)
0620 08:54:41.139796 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.139809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.139898 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.140098 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.140100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.140288 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:41.140382 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.140387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.141089 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.141092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.141654 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.141661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.142060 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.142063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.142154 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.142623 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:41.142636 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:41.143049 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:41.144731 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.144734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.144736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.144962 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.145220 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.145223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.145225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.145301 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.145666 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:41.145669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:41.145758 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.145862 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.146001 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:41.158252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:41.264704 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:41.264945 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:41.264986 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:41.269065 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:41.269129 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:41.269167 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:54:41.269204 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:41.274758 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:41.275848 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:41.276852 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:41.287130 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:41.287130 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:41.287138 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:41.287531 1862 server_base.cc:1034] running on GCE node
I20260620 08:54:41.287930 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:41.288533 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:41.289695 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781945681289676 us; error 42 us; skew 500 ppm
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:41.295197 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:41.296113 1862 webserver.cc:460] Webserver started at http://127.0.98.3:38227/ using document root <none> and password file <none>
I20260620 08:54:41.296649 1862 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:41.296712 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:41.296903 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:41.298907 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "74577fce749b4843b4aa7d3e124a0181"
format_stamp: "Formatted at 2026-06-20 08:54:41 on dist-test-slave-fb1v"
server_key: "8819bcc54c66d274130cb59cdb1f4793"
server_key_iv: "2fcb15ae1c57efe9a6042bbb430e7929"
server_key_version: "encryptionkey@0"
I20260620 08:54:41.299432 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74577fce749b4843b4aa7d3e124a0181"
format_stamp: "Formatted at 2026-06-20 08:54:41 on dist-test-slave-fb1v"
server_key: "8819bcc54c66d274130cb59cdb1f4793"
server_key_iv: "2fcb15ae1c57efe9a6042bbb430e7929"
server_key_version: "encryptionkey@0"
I20260620 08:54:41.303201 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 08:54:41.305673 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:41.306855 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:41.306959 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "74577fce749b4843b4aa7d3e124a0181"
format_stamp: "Formatted at 2026-06-20 08:54:41 on dist-test-slave-fb1v"
server_key: "8819bcc54c66d274130cb59cdb1f4793"
server_key_iv: "2fcb15ae1c57efe9a6042bbb430e7929"
server_key_version: "encryptionkey@0"
I20260620 08:54:41.307053 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:41.316357 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:41.319043 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:41.319197 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:41.320534 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:41.320581 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:41.320629 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:41.320655 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:41.330622 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40929
I20260620 08:54:41.330658 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40929 every 8 connection(s)
I20260620 08:54:41.331693 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:41.333544 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.321250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42817 (local address 127.0.98.62:41393)
0620 08:54:41.321359 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.321363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.322332 (+ 969us) server_negotiation.cc:407] Connection header received
0620 08:54:41.323309 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.323313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.323371 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.323427 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.324730 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.325301 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.326425 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.326675 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.326775 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.329906 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.329910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.329912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.329941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.331557 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.332114 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.332117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.332119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.332208 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.332560 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.332563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.332565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.332743 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.332829 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.333276 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.333392 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:54:41.333793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 1862
I20260620 08:54:41.333906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:41.334149 392 external_mini_cluster.cc:1428] Setting key a23396ef664cf85e39269fb6f1356db9
I20260620 08:54:41.334306 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.321741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.3:42817)
0620 08:54:41.322184 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.322231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.323065 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.323811 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.323818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.324105 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:41.324582 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.324602 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.325414 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.325418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.326313 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.326322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.326821 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.326825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.326992 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.327525 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:41.327544 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:41.329719 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:41.331688 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.331693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.331704 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.331988 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.332317 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.332324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.332326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.332477 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.332841 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:41.332847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:41.333055 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.333210 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.333498 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260620 08:54:41.335331 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41393
I20260620 08:54:41.335457 1993 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:41.335983 1993 heartbeater.cc:511] Master 127.0.98.62:41393 requested a full tablet report, sending...
I20260620 08:54:41.337015 1510 ts_manager.cc:194] Registered new tserver with Master: 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929)
I20260620 08:54:41.337736 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42817
I20260620 08:54:41.344152 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.337988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37659 (local address 127.0.98.62:41393)
0620 08:54:41.338088 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.338092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.338104 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:54:41.338135 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.338138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.338189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.338267 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.338665 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.339143 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.339988 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.340214 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.340298 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.341291 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.341295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.341298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.341322 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.342679 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.343065 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.343069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.343071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.343149 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.343388 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.343391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.343393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.343553 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.343679 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.343912 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.344008 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:54:41.344162 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.337431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.3:37659)
0620 08:54:41.337670 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.337681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.337747 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.338281 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.338284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.338468 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:41.338560 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.338566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.339263 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.339266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.339887 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.339895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.340308 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.340311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.340397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.340843 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:41.340855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:41.341151 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:41.342798 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.342801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.342803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.342978 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.343235 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.343238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.343240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.343311 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.343655 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:41.343658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:41.343765 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.343851 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.343979 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
I20260620 08:54:41.348264 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:41.374125 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.366728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.0.98.62:41393)
0620 08:54:41.366855 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.366859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.366870 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:54:41.366906 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.366909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.366944 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.367010 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.367652 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.368156 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.368973 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.369181 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.369275 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.371424 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.371427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.371429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.371453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.372702 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.373053 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.373057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.373059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.373134 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.373367 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.373370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.373372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.373533 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.373645 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.373863 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.373964 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:54:41.377337 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:54:41.380352 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:41.394425 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.388535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33033 (local address 127.0.0.1:46916)
0620 08:54:41.389294 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.389344 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.389463 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.390192 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.390198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.390219 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.390352 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.390359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.391743 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.391747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.393061 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.393071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.393943 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.393947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.394045 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.394080 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.394179 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.394311 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":59,"spinlock_wait_cycles":384,"thread_start_us":75,"threads_started":1}
I20260620 08:54:41.394730 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.388901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.0.1:45300)
0620 08:54:41.389808 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.389819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.389891 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.390481 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.390484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.390499 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.390564 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.390568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.391545 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.391548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.393069 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.393078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.394239 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.394243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.394417 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.394453 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.394508 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.394604 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":139,"thread_start_us":78,"threads_started":1}
I20260620 08:54:41.394850 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.388868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.0.98.2:37091)
0620 08:54:41.390107 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.390117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.390148 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:54:41.390215 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.390219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.390395 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.390525 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.390900 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.391978 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.393090 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.393917 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.394017 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.394178 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.394625 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.394725 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":102,"threads_started":1}
I20260620 08:54:41.395074 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.389435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.0.98.1:33033)
0620 08:54:41.389733 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.389744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.389767 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:54:41.389842 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.389847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.390047 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.390198 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.390477 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.391651 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.393198 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.393841 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.393934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.394085 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.394642 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.394740 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
I20260620 08:54:41.395123 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.389691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.0.98.3:40929)
0620 08:54:41.389976 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.389984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.390002 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:54:41.390058 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.390062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.390229 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.390371 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.390659 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.391444 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.393205 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.393916 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.394017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.394177 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.394903 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.394996 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
I20260620 08:54:41.395417 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.388674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37091 (local address 127.0.0.1:59576)
0620 08:54:41.389294 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.389344 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.389463 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.390650 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.390653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.390670 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.390777 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.390783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.392084 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.392087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.392980 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.392989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.394239 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.394243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.394392 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.394441 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.394508 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.395299 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":727,"spinlock_wait_cycles":11392,"thread_start_us":98,"threads_started":1}
I20260620 08:54:41.397919 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 432aff19344840799dd29529aa13dfb4 (DEFAULT_TABLE table=test-table [id=fce023ea54c74138bd784653ec62dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:41.398049 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 432aff19344840799dd29529aa13dfb4 (DEFAULT_TABLE table=test-table [id=fce023ea54c74138bd784653ec62dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:41.398048 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 432aff19344840799dd29529aa13dfb4 (DEFAULT_TABLE table=test-table [id=fce023ea54c74138bd784653ec62dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:41.398902 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432aff19344840799dd29529aa13dfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.398995 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432aff19344840799dd29529aa13dfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.398995 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432aff19344840799dd29529aa13dfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.405025 2016 tablet_bootstrap.cc:492] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Bootstrap starting.
I20260620 08:54:41.406163 2018 tablet_bootstrap.cc:492] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Bootstrap starting.
I20260620 08:54:41.406809 2017 tablet_bootstrap.cc:492] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Bootstrap starting.
I20260620 08:54:41.407235 2016 tablet_bootstrap.cc:654] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.408190 2016 log.cc:824] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:41.408227 2018 tablet_bootstrap.cc:654] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.408550 2017 tablet_bootstrap.cc:654] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.409000 2018 log.cc:824] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:41.409292 2017 log.cc:824] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:41.410257 2016 tablet_bootstrap.cc:492] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: No bootstrap required, opened a new log
I20260620 08:54:41.410451 2016 ts_tablet_manager.cc:1397] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:54:41.410791 2018 tablet_bootstrap.cc:492] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: No bootstrap required, opened a new log
I20260620 08:54:41.410964 2018 ts_tablet_manager.cc:1397] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:54:41.411165 2017 tablet_bootstrap.cc:492] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: No bootstrap required, opened a new log
I20260620 08:54:41.411332 2017 ts_tablet_manager.cc:1397] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:54:41.413434 2016 raft_consensus.cc:348] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.413648 2016 raft_consensus.cc:374] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.413739 2016 raft_consensus.cc:729] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74577fce749b4843b4aa7d3e124a0181, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.414166 2016 consensus_queue.cc:260] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.415009 2016 ts_tablet_manager.cc:1428] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:54:41.415000 2017 raft_consensus.cc:348] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.415129 2018 raft_consensus.cc:348] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.415306 2017 raft_consensus.cc:374] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.415334 1993 heartbeater.cc:503] Master 127.0.98.62:41393 was elected leader, sending a full tablet report...
I20260620 08:54:41.415354 2018 raft_consensus.cc:374] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.415406 2018 raft_consensus.cc:729] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e734f57eac6949d88ac64a0753d248c1, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.415433 2017 raft_consensus.cc:729] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b995c465d34fa58cd9d703d969ee79, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.415858 2018 consensus_queue.cc:260] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.415858 2017 consensus_queue.cc:260] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.416652 1853 heartbeater.cc:503] Master 127.0.98.62:41393 was elected leader, sending a full tablet report...
I20260620 08:54:41.416764 2017 ts_tablet_manager.cc:1428] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 08:54:41.416812 2018 ts_tablet_manager.cc:1428] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 08:54:41.417584 1713 heartbeater.cc:503] Master 127.0.98.62:41393 was elected leader, sending a full tablet report...
I20260620 08:54:41.421149 2023 raft_consensus.cc:482] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:41.421320 2023 raft_consensus.cc:504] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.422317 2023 leader_election.cc:288] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033), 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929)
I20260620 08:54:41.425870 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.422547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.98.2:39589)
0620 08:54:41.422659 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.422676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.422766 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.423013 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.423017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.423032 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.423156 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.423162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.423967 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.423971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.424781 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.424790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.425430 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.425434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.425545 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.425571 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.425645 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.425727 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 08:54:41.425870 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.422598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39589 (local address 127.0.98.3:40929)
0620 08:54:41.422711 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.422716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.422730 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:54:41.422877 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.422881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.422926 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.422982 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.423254 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.423866 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.424906 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.425320 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.425420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.425488 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.425667 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.425736 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 08:54:41.425985 1714 tablet.cc:2369] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:41.426914 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.422689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53471 (local address 127.0.98.1:33033)
0620 08:54:41.422797 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.422801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.423076 (+ 275us) server_negotiation.cc:407] Connection header received
0620 08:54:41.423170 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.423173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.423211 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.423269 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.423982 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.424469 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.425406 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.425787 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.425878 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.425933 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.426763 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.426832 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:54:41.426936 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.422547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33033 (local address 127.0.98.2:53471)
0620 08:54:41.422995 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.423007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.423095 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.423282 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.423284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.423298 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.423879 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.423888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.424560 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.424564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.425299 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.425306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.426569 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.426572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.426641 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.426659 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.426722 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.426786 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":179,"thread_start_us":95,"threads_started":1}
I20260620 08:54:41.426966 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432aff19344840799dd29529aa13dfb4" candidate_uuid: "54b995c465d34fa58cd9d703d969ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74577fce749b4843b4aa7d3e124a0181" is_pre_election: true
I20260620 08:54:41.427214 1947 raft_consensus.cc:2454] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b995c465d34fa58cd9d703d969ee79 in term 0.
I20260620 08:54:41.427886 1741 leader_election.cc:302] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b995c465d34fa58cd9d703d969ee79, 74577fce749b4843b4aa7d3e124a0181; no voters:
I20260620 08:54:41.428129 2023 raft_consensus.cc:2781] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:41.428191 2023 raft_consensus.cc:482] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:41.428112 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432aff19344840799dd29529aa13dfb4" candidate_uuid: "54b995c465d34fa58cd9d703d969ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e734f57eac6949d88ac64a0753d248c1" is_pre_election: true
I20260620 08:54:41.428220 2023 raft_consensus.cc:3037] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.428368 1667 raft_consensus.cc:2454] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b995c465d34fa58cd9d703d969ee79 in term 0.
I20260620 08:54:41.429257 2023 raft_consensus.cc:504] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.429812 2023 leader_election.cc:288] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [CANDIDATE]: Term 1 election: Requested vote from peers e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033), 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929)
I20260620 08:54:41.430120 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432aff19344840799dd29529aa13dfb4" candidate_uuid: "54b995c465d34fa58cd9d703d969ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74577fce749b4843b4aa7d3e124a0181"
I20260620 08:54:41.430121 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432aff19344840799dd29529aa13dfb4" candidate_uuid: "54b995c465d34fa58cd9d703d969ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e734f57eac6949d88ac64a0753d248c1"
I20260620 08:54:41.430224 1947 raft_consensus.cc:3037] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.430238 1667 raft_consensus.cc:3037] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.431252 1947 raft_consensus.cc:2454] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b995c465d34fa58cd9d703d969ee79 in term 1.
I20260620 08:54:41.431262 1667 raft_consensus.cc:2454] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b995c465d34fa58cd9d703d969ee79 in term 1.
I20260620 08:54:41.431561 1739 leader_election.cc:302] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b995c465d34fa58cd9d703d969ee79, e734f57eac6949d88ac64a0753d248c1; no voters:
I20260620 08:54:41.431739 2023 raft_consensus.cc:2781] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:41.431942 2023 raft_consensus.cc:686] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 LEADER]: Becoming Leader. State: Replica: 54b995c465d34fa58cd9d703d969ee79, State: Running, Role: LEADER
I20260620 08:54:41.432257 2023 consensus_queue.cc:237] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.435130 1508 catalog_manager.cc:5495] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "54b995c465d34fa58cd9d703d969ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } health_report { overall_health: UNKNOWN } } }
I20260620 08:54:41.453656 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.449996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.0.98.2:37091)
0620 08:54:41.450102 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.450105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.450136 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:54:41.450301 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.450304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.450358 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.450435 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:41.450702 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.451217 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.452197 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.452432 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.452519 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.452821 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.453297 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.453383 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.453487 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.453541 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:54:41.459163 1667 raft_consensus.cc:1261] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Refusing update from remote peer 54b995c465d34fa58cd9d703d969ee79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:41.459446 1947 raft_consensus.cc:1261] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Refusing update from remote peer 54b995c465d34fa58cd9d703d969ee79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:41.459843 2026 consensus_queue.cc:1035] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:41.460004 2023 consensus_queue.cc:1035] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:41.467494 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298849511250317312 to 7298849511148859392. Current Snapshot: MvccSnapshot[applied={T|T < 7298849511250317312}]
I20260620 08:54:41.478469 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:41.478634 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:41.481051 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: Sending DeleteTablet for 3 replicas of tablet 432aff19344840799dd29529aa13dfb4
I20260620 08:54:41.481813 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 432aff19344840799dd29529aa13dfb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46916
I20260620 08:54:41.481813 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 432aff19344840799dd29529aa13dfb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59576
I20260620 08:54:41.481930 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 432aff19344840799dd29529aa13dfb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45300
I20260620 08:54:41.482465 2038 tablet_replica.cc:330] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: stopping tablet replica
I20260620 08:54:41.482551 2039 tablet_replica.cc:330] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: stopping tablet replica
I20260620 08:54:41.482647 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:54:41.482774 2038 raft_consensus.cc:2229] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:41.482772 2039 raft_consensus.cc:2229] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 08:54:41.483042 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:41.483121 2039 raft_consensus.cc:2258] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:41.483671 2038 raft_consensus.cc:2258] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:41.484752 2039 ts_tablet_manager.cc:1905] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:41.485131 2038 ts_tablet_manager.cc:1905] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:41.487389 2039 ts_tablet_manager.cc:1918] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:41.487484 2039 log.cc:1196] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/432aff19344840799dd29529aa13dfb4
I20260620 08:54:41.487787 2039 ts_tablet_manager.cc:1939] T 432aff19344840799dd29529aa13dfb4 P e734f57eac6949d88ac64a0753d248c1: Deleting consensus metadata
I20260620 08:54:41.488893 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 68a1e95f713f4d9ebb8794a91898508a (DEFAULT_TABLE table=test-table [id=153ab792290f48a78d7bbe248589c1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:41.488971 1495 catalog_manager.cc:4841] TS e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033): tablet 432aff19344840799dd29529aa13dfb4 (table test-table [id=fce023ea54c74138bd784653ec62dcd8]) successfully deleted
I20260620 08:54:41.489123 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a1e95f713f4d9ebb8794a91898508a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.489116 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 68a1e95f713f4d9ebb8794a91898508a (DEFAULT_TABLE table=test-table [id=153ab792290f48a78d7bbe248589c1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:41.489370 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a1e95f713f4d9ebb8794a91898508a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.489665 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 68a1e95f713f4d9ebb8794a91898508a (DEFAULT_TABLE table=test-table [id=153ab792290f48a78d7bbe248589c1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:41.489912 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a1e95f713f4d9ebb8794a91898508a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.491439 2017 tablet_bootstrap.cc:492] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Bootstrap starting.
I20260620 08:54:41.491639 2018 tablet_bootstrap.cc:492] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Bootstrap starting.
I20260620 08:54:41.492225 2016 tablet_bootstrap.cc:492] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Bootstrap starting.
I20260620 08:54:41.492266 2017 tablet_bootstrap.cc:654] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.492509 2018 tablet_bootstrap.cc:654] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.493119 2016 tablet_bootstrap.cc:654] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.493139 2038 ts_tablet_manager.cc:1918] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:41.493221 2038 log.cc:1196] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/432aff19344840799dd29529aa13dfb4
I20260620 08:54:41.493595 2038 ts_tablet_manager.cc:1939] T 432aff19344840799dd29529aa13dfb4 P 54b995c465d34fa58cd9d703d969ee79: Deleting consensus metadata
I20260620 08:54:41.493845 2040 tablet_replica.cc:330] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: stopping tablet replica
I20260620 08:54:41.494155 2040 raft_consensus.cc:2229] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:41.494472 2040 raft_consensus.cc:2258] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:41.494735 1496 catalog_manager.cc:4841] TS 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091): tablet 432aff19344840799dd29529aa13dfb4 (table test-table [id=fce023ea54c74138bd784653ec62dcd8]) successfully deleted
I20260620 08:54:41.496047 2018 tablet_bootstrap.cc:492] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: No bootstrap required, opened a new log
I20260620 08:54:41.496131 2018 ts_tablet_manager.cc:1397] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:54:41.496233 2016 tablet_bootstrap.cc:492] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: No bootstrap required, opened a new log
I20260620 08:54:41.496264 2017 tablet_bootstrap.cc:492] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: No bootstrap required, opened a new log
I20260620 08:54:41.496311 2016 ts_tablet_manager.cc:1397] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 08:54:41.496359 2017 ts_tablet_manager.cc:1397] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:54:41.496562 2018 raft_consensus.cc:348] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.496665 2018 raft_consensus.cc:374] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.496694 2018 raft_consensus.cc:729] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e734f57eac6949d88ac64a0753d248c1, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.496697 2016 raft_consensus.cc:348] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.496788 2016 raft_consensus.cc:374] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.496815 2016 raft_consensus.cc:729] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74577fce749b4843b4aa7d3e124a0181, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.496812 2018 consensus_queue.cc:260] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.496846 2017 raft_consensus.cc:348] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.496928 2017 raft_consensus.cc:374] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.496955 2017 raft_consensus.cc:729] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b995c465d34fa58cd9d703d969ee79, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.496918 2016 consensus_queue.cc:260] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.497063 2018 ts_tablet_manager.cc:1428] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:54:41.497058 2017 consensus_queue.cc:260] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.497134 2016 ts_tablet_manager.cc:1428] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:41.497270 2017 ts_tablet_manager.cc:1428] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:41.498557 2040 ts_tablet_manager.cc:1905] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:41.501112 2040 ts_tablet_manager.cc:1918] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:41.501181 2040 log.cc:1196] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/432aff19344840799dd29529aa13dfb4
I20260620 08:54:41.501407 2040 ts_tablet_manager.cc:1939] T 432aff19344840799dd29529aa13dfb4 P 74577fce749b4843b4aa7d3e124a0181: Deleting consensus metadata
I20260620 08:54:41.502435 1497 catalog_manager.cc:4841] TS 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929): tablet 432aff19344840799dd29529aa13dfb4 (table test-table [id=fce023ea54c74138bd784653ec62dcd8]) successfully deleted
I20260620 08:54:41.551183 2022 raft_consensus.cc:482] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:41.551322 2022 raft_consensus.cc:504] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.552448 2022 leader_election.cc:288] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033), 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091)
I20260620 08:54:41.556847 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.552875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42511 (local address 127.0.98.2:37091)
0620 08:54:41.553022 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.553026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.553341 (+ 315us) server_negotiation.cc:407] Connection header received
0620 08:54:41.553502 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.553505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.553551 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.553617 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.553856 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.554526 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.555682 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.556195 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.556268 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.556337 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.556637 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.556722 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 08:54:41.556947 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.552650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33033 (local address 127.0.98.3:56695)
0620 08:54:41.552768 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.552783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.552867 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.553264 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.553268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.553290 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.553446 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.553453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.554418 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.554421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.555694 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.555705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.556587 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.556590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.556686 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.556705 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.556772 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.556837 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":25}
I20260620 08:54:41.556989 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.552890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56695 (local address 127.0.98.1:33033)
0620 08:54:41.553021 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.553025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.553038 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:41.553089 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.553092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.553137 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.553228 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.553560 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.554320 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.555834 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.556473 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.556577 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.556636 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.556791 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.556859 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:54:41.557299 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.552663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37091 (local address 127.0.98.3:42511)
0620 08:54:41.553273 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.553286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.553405 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.553635 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.553638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.553652 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.553758 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.553764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.554626 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.554630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.555512 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.555522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.556426 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.556430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.556508 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.556526 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.557111 (+ 585us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.557176 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":278,"thread_start_us":69,"threads_started":1}
I20260620 08:54:41.557533 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a1e95f713f4d9ebb8794a91898508a" candidate_uuid: "74577fce749b4843b4aa7d3e124a0181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e734f57eac6949d88ac64a0753d248c1" is_pre_election: true
I20260620 08:54:41.557682 1667 raft_consensus.cc:2454] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74577fce749b4843b4aa7d3e124a0181 in term 0.
I20260620 08:54:41.557874 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a1e95f713f4d9ebb8794a91898508a" candidate_uuid: "74577fce749b4843b4aa7d3e124a0181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b995c465d34fa58cd9d703d969ee79" is_pre_election: true
I20260620 08:54:41.557996 1879 leader_election.cc:302] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74577fce749b4843b4aa7d3e124a0181, e734f57eac6949d88ac64a0753d248c1; no voters:
I20260620 08:54:41.558099 1807 raft_consensus.cc:2454] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74577fce749b4843b4aa7d3e124a0181 in term 0.
I20260620 08:54:41.558247 2022 raft_consensus.cc:2781] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:41.558300 2022 raft_consensus.cc:482] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:41.558347 2022 raft_consensus.cc:3037] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.559525 2022 raft_consensus.cc:504] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.559921 2022 leader_election.cc:288] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [CANDIDATE]: Term 1 election: Requested vote from peers e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033), 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091)
I20260620 08:54:41.560551 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a1e95f713f4d9ebb8794a91898508a" candidate_uuid: "74577fce749b4843b4aa7d3e124a0181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e734f57eac6949d88ac64a0753d248c1"
I20260620 08:54:41.560576 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a1e95f713f4d9ebb8794a91898508a" candidate_uuid: "74577fce749b4843b4aa7d3e124a0181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b995c465d34fa58cd9d703d969ee79"
I20260620 08:54:41.560704 1667 raft_consensus.cc:3037] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.560704 1807 raft_consensus.cc:3037] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.562099 1807 raft_consensus.cc:2454] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74577fce749b4843b4aa7d3e124a0181 in term 1.
I20260620 08:54:41.562345 1667 raft_consensus.cc:2454] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74577fce749b4843b4aa7d3e124a0181 in term 1.
I20260620 08:54:41.562434 1880 leader_election.cc:302] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b995c465d34fa58cd9d703d969ee79, 74577fce749b4843b4aa7d3e124a0181; no voters:
I20260620 08:54:41.562693 2022 raft_consensus.cc:2781] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:41.562841 2022 raft_consensus.cc:686] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 1 LEADER]: Becoming Leader. State: Replica: 74577fce749b4843b4aa7d3e124a0181, State: Running, Role: LEADER
I20260620 08:54:41.563072 2022 consensus_queue.cc:237] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.565254 1510 catalog_manager.cc:5495] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74577fce749b4843b4aa7d3e124a0181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } health_report { overall_health: HEALTHY } } }
W20260620 08:54:41.582703 1994 tablet.cc:2369] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:41.598222 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.589633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.0.1:43992)
0620 08:54:41.589919 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.589930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.590036 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.590267 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.590270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.590646 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:41.590739 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.590754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.591543 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.591546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.592157 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.592165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.592580 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.592583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.592688 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.593222 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:41.593234 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:41.594831 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:41.596660 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.596668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.596672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.596921 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.597232 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:41.597235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:41.597236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:41.597320 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:41.597773 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:41.597779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:41.597884 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.597977 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.598086 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":17,"thread_start_us":124,"threads_started":1}
I20260620 08:54:41.598364 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.589661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.0.98.62:41393)
0620 08:54:41.589773 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.589777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.589997 (+ 220us) server_negotiation.cc:407] Connection header received
0620 08:54:41.590143 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.590146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.590183 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.590241 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.590854 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.591448 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.592269 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.592487 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.592570 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.594975 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.594979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.594982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.595008 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.596556 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.597026 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.597030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.597032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.597120 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.597412 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.597416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.597418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.597633 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.597734 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.598057 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.598175 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":37}
I20260620 08:54:41.602445 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:54:41.603377 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:41.608090 1787 tablet_service.cc:1467] Processing CreateTablet for tablet c77148cf2c81464c9b3947f0c5b1a6c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f0aa6f51f3e47f2aab219c545a311e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:54:41.608093 1647 tablet_service.cc:1467] Processing CreateTablet for tablet c77148cf2c81464c9b3947f0c5b1a6c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f0aa6f51f3e47f2aab219c545a311e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:54:41.608446 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77148cf2c81464c9b3947f0c5b1a6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.608497 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77148cf2c81464c9b3947f0c5b1a6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.610781 2017 tablet_bootstrap.cc:492] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: Bootstrap starting.
I20260620 08:54:41.611567 2018 tablet_bootstrap.cc:492] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: Bootstrap starting.
I20260620 08:54:41.611697 2017 tablet_bootstrap.cc:654] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.612725 2018 tablet_bootstrap.cc:654] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.613245 1927 tablet_service.cc:1467] Processing CreateTablet for tablet c77148cf2c81464c9b3947f0c5b1a6c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f0aa6f51f3e47f2aab219c545a311e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:54:41.613552 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77148cf2c81464c9b3947f0c5b1a6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:41.615885 2016 tablet_bootstrap.cc:492] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: Bootstrap starting.
I20260620 08:54:41.616412 2017 tablet_bootstrap.cc:492] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: No bootstrap required, opened a new log
I20260620 08:54:41.616492 2017 ts_tablet_manager.cc:1397] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260620 08:54:41.616806 2016 tablet_bootstrap.cc:654] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:41.616882 2017 raft_consensus.cc:348] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.616993 2017 raft_consensus.cc:374] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.617027 2017 raft_consensus.cc:729] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b995c465d34fa58cd9d703d969ee79, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.617143 2017 consensus_queue.cc:260] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.617353 2026 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.617476 2026 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:41.617702 2017 ts_tablet_manager.cc:1428] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:54:41.618456 2018 tablet_bootstrap.cc:492] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: No bootstrap required, opened a new log
I20260620 08:54:41.618474 2016 tablet_bootstrap.cc:492] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: No bootstrap required, opened a new log
I20260620 08:54:41.618533 2018 ts_tablet_manager.cc:1397] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260620 08:54:41.618541 2016 ts_tablet_manager.cc:1397] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:54:41.618894 2016 raft_consensus.cc:348] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.618981 2016 raft_consensus.cc:374] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.619006 2016 raft_consensus.cc:729] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74577fce749b4843b4aa7d3e124a0181, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.618955 2018 raft_consensus.cc:348] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.619063 2018 raft_consensus.cc:374] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:41.619098 2018 raft_consensus.cc:729] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e734f57eac6949d88ac64a0753d248c1, State: Initialized, Role: FOLLOWER
I20260620 08:54:41.619181 2016 consensus_queue.cc:260] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.619215 2018 consensus_queue.cc:260] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.619436 2016 ts_tablet_manager.cc:1428] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:41.619469 2018 ts_tablet_manager.cc:1428] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:54:41.619899 2022 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.620015 2022 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:41.620047 2024 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.620150 2024 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 08:54:41.633106 1854 tablet.cc:2369] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:41.690863 2024 raft_consensus.cc:482] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:41.691011 2024 raft_consensus.cc:504] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.692189 2024 leader_election.cc:288] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091), 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929)
I20260620 08:54:41.696545 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.692858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57681 (local address 127.0.98.3:40929)
0620 08:54:41.692979 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.692984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.692997 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:41.693041 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.693043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.693078 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.693136 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.693359 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.694095 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.695381 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.695984 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.696061 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.696110 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.696359 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.696429 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:54:41.696568 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.692420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.98.1:57681)
0620 08:54:41.692844 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.692858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.692962 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.693151 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.693153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.693167 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.693255 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.693261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.694229 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.694232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.695271 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.695280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.696098 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.696102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.696205 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.696231 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.696317 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.696401 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260620 08:54:41.696995 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.692515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54303 (local address 127.0.98.2:37091)
0620 08:54:41.692649 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.692654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.693367 (+ 713us) server_negotiation.cc:407] Connection header received
0620 08:54:41.693457 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.693460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.693504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.693562 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:41.693894 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.694654 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.695898 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.696428 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.696521 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.696581 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.696798 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.696881 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:54:41.697183 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77148cf2c81464c9b3947f0c5b1a6c4" candidate_uuid: "e734f57eac6949d88ac64a0753d248c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74577fce749b4843b4aa7d3e124a0181" is_pre_election: true
I20260620 08:54:41.697328 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.692423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37091 (local address 127.0.98.1:54303)
0620 08:54:41.693281 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.693292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.693380 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.693640 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.693643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.693660 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:41.693769 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.693774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.694766 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.694769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.695753 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.695763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.696548 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.696552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.696650 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.696674 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.697118 (+ 444us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.697186 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":53,"thread_start_us":125,"threads_started":1}
I20260620 08:54:41.697335 1947 raft_consensus.cc:2454] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e734f57eac6949d88ac64a0753d248c1 in term 0.
I20260620 08:54:41.697676 1601 leader_election.cc:302] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74577fce749b4843b4aa7d3e124a0181, e734f57eac6949d88ac64a0753d248c1; no voters:
I20260620 08:54:41.697743 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77148cf2c81464c9b3947f0c5b1a6c4" candidate_uuid: "e734f57eac6949d88ac64a0753d248c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b995c465d34fa58cd9d703d969ee79" is_pre_election: true
I20260620 08:54:41.697871 1807 raft_consensus.cc:2454] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e734f57eac6949d88ac64a0753d248c1 in term 0.
I20260620 08:54:41.697940 2024 raft_consensus.cc:2781] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:41.698004 2024 raft_consensus.cc:482] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:41.698051 2024 raft_consensus.cc:3037] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.699054 2024 raft_consensus.cc:504] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.699359 2024 leader_election.cc:288] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [CANDIDATE]: Term 1 election: Requested vote from peers 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091), 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929)
I20260620 08:54:41.699672 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77148cf2c81464c9b3947f0c5b1a6c4" candidate_uuid: "e734f57eac6949d88ac64a0753d248c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b995c465d34fa58cd9d703d969ee79"
I20260620 08:54:41.699810 1807 raft_consensus.cc:3037] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.699851 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77148cf2c81464c9b3947f0c5b1a6c4" candidate_uuid: "e734f57eac6949d88ac64a0753d248c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74577fce749b4843b4aa7d3e124a0181"
I20260620 08:54:41.699957 1947 raft_consensus.cc:3037] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:41.700922 1947 raft_consensus.cc:2454] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e734f57eac6949d88ac64a0753d248c1 in term 1.
I20260620 08:54:41.700922 1807 raft_consensus.cc:2454] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e734f57eac6949d88ac64a0753d248c1 in term 1.
I20260620 08:54:41.701180 1600 leader_election.cc:302] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b995c465d34fa58cd9d703d969ee79, e734f57eac6949d88ac64a0753d248c1; no voters:
I20260620 08:54:41.701340 2024 raft_consensus.cc:2781] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:41.701527 2024 raft_consensus.cc:686] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [term 1 LEADER]: Becoming Leader. State: Replica: e734f57eac6949d88ac64a0753d248c1, State: Running, Role: LEADER
I20260620 08:54:41.701753 2024 consensus_queue.cc:237] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } }
I20260620 08:54:41.702937 2058 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: TxnStatusTablet state changed. Reason: New leader e734f57eac6949d88ac64a0753d248c1. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.703064 2058 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:54:41.703737 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:54:41.703920 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:54:41.704224 1510 catalog_manager.cc:5495] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e734f57eac6949d88ac64a0753d248c1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } health_report { overall_health: UNKNOWN } } }
I20260620 08:54:41.720324 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.717004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33033 (local address 127.0.0.1:46920)
0620 08:54:41.717137 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.717160 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.717254 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.717527 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.717530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.717537 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:41.717630 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.717635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.718309 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.718312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.718900 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.718907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.719300 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.719303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.719385 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.719425 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.720047 (+ 622us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.720060 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.720119 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.720201 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:54:41.720321 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.717171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.0.98.1:33033)
0620 08:54:41.717299 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.717304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.717319 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:54:41.717382 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.717384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.717431 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.717506 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:41.717718 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.718209 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.719000 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.719206 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.719287 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.719549 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.719961 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.720057 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.720138 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.720188 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:54:41.723258 1947 raft_consensus.cc:1261] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Refusing update from remote peer e734f57eac6949d88ac64a0753d248c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:41.723373 1807 raft_consensus.cc:1261] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Refusing update from remote peer e734f57eac6949d88ac64a0753d248c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:41.723912 2058 consensus_queue.cc:1035] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:41.724077 2024 consensus_queue.cc:1035] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:41.725507 2026 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: TxnStatusTablet state changed. Reason: New leader e734f57eac6949d88ac64a0753d248c1. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.725512 2022 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: TxnStatusTablet state changed. Reason: New leader e734f57eac6949d88ac64a0753d248c1. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.725620 2022 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:41.725620 2026 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:41.727490 2058 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.727599 2058 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:54:41.727871 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298849512333512704 to 7298849512252588032. Current Snapshot: MvccSnapshot[applied={T|T < 7298849512333512704}]
I20260620 08:54:41.728053 2026 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.728143 2026 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P 54b995c465d34fa58cd9d703d969ee79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:41.729598 2024 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.729686 2024 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P e734f57eac6949d88ac64a0753d248c1: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:54:41.730599 2022 tablet_replica.cc:439] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e734f57eac6949d88ac64a0753d248c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 } } peers { permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } } peers { permanent_uuid: "74577fce749b4843b4aa7d3e124a0181" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } }
I20260620 08:54:41.730711 2022 tablet_replica.cc:442] T c77148cf2c81464c9b3947f0c5b1a6c4 P 74577fce749b4843b4aa7d3e124a0181: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:41.731016 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298849512333512704 to 7298849512252588032. Current Snapshot: MvccSnapshot[applied={T|T < 7298849512333512704 or (T in {7298849512333512704})}]
I20260620 08:54:41.738458 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.734659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.0.98.3:40929)
0620 08:54:41.734756 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.734760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.734773 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:41.734953 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.734955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.735005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.735096 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:41.735309 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.735876 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.737057 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.737331 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.737416 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.737676 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.738099 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.738181 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.738255 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.738299 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 08:54:41.748534 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.745283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47693 (local address 127.0.98.1:33033)
0620 08:54:41.745413 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.745417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.745431 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:54:41.745478 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.745481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.745524 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.745583 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:41.745820 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.746341 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.747532 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.747780 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.747866 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.748170 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.748239 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.748300 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.748383 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.748429 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:54:41.748590 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.745164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33033 (local address 127.0.98.3:47693)
0620 08:54:41.745302 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.745316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.745398 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.745602 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.745605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.745613 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:41.745729 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.745733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.746452 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.746455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.747415 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.747425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.747876 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.747880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.747975 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.748024 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.748311 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.748317 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.748376 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.748456 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:54:41.755275 1807 raft_consensus.cc:1261] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Refusing update from remote peer 74577fce749b4843b4aa7d3e124a0181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:41.755299 1667 raft_consensus.cc:1261] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Refusing update from remote peer 74577fce749b4843b4aa7d3e124a0181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:41.755739 2045 consensus_queue.cc:1035] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b995c465d34fa58cd9d703d969ee79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:41.755904 2022 consensus_queue.cc:1035] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e734f57eac6949d88ac64a0753d248c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:41.776749 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.772895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59467 (local address 127.0.98.3:40929)
0620 08:54:41.773095 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.773098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.773202 (+ 104us) server_negotiation.cc:407] Connection header received
0620 08:54:41.773362 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.773365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.773421 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.773494 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:41.773727 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.774507 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.775393 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.775749 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.775840 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.776288 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.776374 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.776434 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.776553 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.776605 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260620 08:54:41.777094 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.772807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.98.1:59467)
0620 08:54:41.773126 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:41.773140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:41.773273 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:41.773507 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:41.773510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:41.773518 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:41.773631 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.773638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.774598 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.774600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.775276 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:41.775286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.775914 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.775918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:41.776015 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.776062 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:41.776449 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:41.776455 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:54:41.776878 (+ 423us) client_negotiation.cc:240] Negotiation successful
0620 08:54:41.776956 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":122,"threads_started":1}
I20260620 08:54:41.799706 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:41.799885 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:41.802076 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9041f14be1954768b58e2921603578e9: Sending DeleteTablet for 3 replicas of tablet 68a1e95f713f4d9ebb8794a91898508a
I20260620 08:54:41.802700 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a1e95f713f4d9ebb8794a91898508a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59576
I20260620 08:54:41.802829 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a1e95f713f4d9ebb8794a91898508a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45300
I20260620 08:54:41.802873 2038 tablet_replica.cc:330] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: stopping tablet replica
I20260620 08:54:41.802978 2038 raft_consensus.cc:2229] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:41.802992 2040 tablet_replica.cc:330] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: stopping tablet replica
I20260620 08:54:41.803076 2038 raft_consensus.cc:2258] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:41.803095 2040 raft_consensus.cc:2229] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:41.803370 2040 raft_consensus.cc:2258] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:41.803849 2038 ts_tablet_manager.cc:1905] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:41.804282 2040 ts_tablet_manager.cc:1905] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:41.806577 2038 ts_tablet_manager.cc:1918] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:41.806643 2038 log.cc:1196] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/68a1e95f713f4d9ebb8794a91898508a
I20260620 08:54:41.806864 2038 ts_tablet_manager.cc:1939] T 68a1e95f713f4d9ebb8794a91898508a P 54b995c465d34fa58cd9d703d969ee79: Deleting consensus metadata
I20260620 08:54:41.806903 2040 ts_tablet_manager.cc:1918] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:41.806968 2040 log.cc:1196] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/68a1e95f713f4d9ebb8794a91898508a
I20260620 08:54:41.807709 1496 catalog_manager.cc:4841] TS 54b995c465d34fa58cd9d703d969ee79 (127.0.98.2:37091): tablet 68a1e95f713f4d9ebb8794a91898508a (table test-table [id=153ab792290f48a78d7bbe248589c1e7]) successfully deleted
I20260620 08:54:41.807212 2040 ts_tablet_manager.cc:1939] T 68a1e95f713f4d9ebb8794a91898508a P 74577fce749b4843b4aa7d3e124a0181: Deleting consensus metadata
I20260620 08:54:41.808069 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a1e95f713f4d9ebb8794a91898508a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46916
I20260620 08:54:41.808246 2039 tablet_replica.cc:330] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: stopping tablet replica
I20260620 08:54:41.808382 2039 raft_consensus.cc:2229] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:41.808496 2039 raft_consensus.cc:2258] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:41.810170 2039 ts_tablet_manager.cc:1905] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:41.812747 2039 ts_tablet_manager.cc:1918] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:41.812822 2039 log.cc:1196] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/68a1e95f713f4d9ebb8794a91898508a
I20260620 08:54:41.813047 2039 ts_tablet_manager.cc:1939] T 68a1e95f713f4d9ebb8794a91898508a P e734f57eac6949d88ac64a0753d248c1: Deleting consensus metadata
I20260620 08:54:41.813736 1495 catalog_manager.cc:4841] TS e734f57eac6949d88ac64a0753d248c1 (127.0.98.1:33033): tablet 68a1e95f713f4d9ebb8794a91898508a (table test-table [id=153ab792290f48a78d7bbe248589c1e7]) successfully deleted
I20260620 08:54:41.814330 1497 catalog_manager.cc:4841] TS 74577fce749b4843b4aa7d3e124a0181 (127.0.98.3:40929): tablet 68a1e95f713f4d9ebb8794a91898508a (table test-table [id=153ab792290f48a78d7bbe248589c1e7]) successfully deleted
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:41.825169 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.809816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.0.98.1:33033)
0620 08:54:41.809924 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.809928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.814431 (+ 4503us) server_negotiation.cc:407] Connection header received
0620 08:54:41.814614 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.814617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.814671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.814752 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.815515 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.816205 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.817379 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.817713 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.817812 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.821350 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.821360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.821376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.821404 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.823329 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.823835 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.823841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.823843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.823931 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.824196 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.824199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.824201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.824474 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.824667 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.824887 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.825005 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:54:41.826131 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46930
I20260620 08:54:41.837829 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:41.828018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.0.98.62:41393)
0620 08:54:41.828143 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:41.828148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:41.828567 (+ 419us) server_negotiation.cc:407] Connection header received
0620 08:54:41.828679 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:41.828683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:41.828736 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:41.828825 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:41.829548 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.830349 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.831895 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:41.832298 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:41.832481 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:41.834193 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:41.834197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:41.834199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:41.834231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:41.836029 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.836561 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.836565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.836567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.836665 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:41.836944 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:41.836947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:41.836949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:41.837176 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:41.837308 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:41.837533 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 08:54:41.837664 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 08:54:41.838197 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43994
I20260620 08:54:41.839638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1582
I20260620 08:54:41.846262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1722
I20260620 08:54:41.853613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1862
I20260620 08:54:41.858464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 1478
2026-06-20T08:54:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5406 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[2082](info): set up 2 sockets
Jun 20 08:54:41 dist-test-slave-fb1v krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260620 08:54:42.803299 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:42 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:42.919437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:43.026446 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:43.026713 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:43.026768 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:43.030671 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:43.030725 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:54:43.030747 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:43.030771 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:43.030793 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:43.030822 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:43.035921 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32791
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:43.036881 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:43.037807 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:43.045773 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.045835 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.045902 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:43.046284 2095 server_base.cc:1034] running on GCE node
I20260620 08:54:43.046778 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:43.047825 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:43.048980 2095 hybrid_clock.cc:648] HybridClock initialized: now 1781945683048966 us; error 28 us; skew 500 ppm
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:43.051494 2095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:43.052380 2095 webserver.cc:460] Webserver started at http://127.0.98.62:45817/ using document root <none> and password file <none>
I20260620 08:54:43.052882 2095 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:43.052938 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:43.053121 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:43.055008 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "0e94064ca2cc424c90ef6f2657ce5469"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "3ba98ef04aa446816d0ecced857e1d00"
server_key_iv: "a56960f5ac0d2328ccf629b92641d9f7"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.055497 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e94064ca2cc424c90ef6f2657ce5469"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "3ba98ef04aa446816d0ecced857e1d00"
server_key_iv: "a56960f5ac0d2328ccf629b92641d9f7"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.058902 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 08:54:43.061041 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.062193 2095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:43.062296 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "0e94064ca2cc424c90ef6f2657ce5469"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "3ba98ef04aa446816d0ecced857e1d00"
server_key_iv: "a56960f5ac0d2328ccf629b92641d9f7"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.062397 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:43.083461 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:43.086145 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:43.086313 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:43.093498 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32791
I20260620 08:54:43.093557 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32791 every 8 connection(s)
I20260620 08:54:43.094405 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:43.095368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 2095
I20260620 08:54:43.095479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:43.095798 392 external_mini_cluster.cc:1428] Setting key 1183a4da608e6cab4724e6c7af54372a
I20260620 08:54:43.097470 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.103567 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Bootstrap starting.
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:43.105867 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.106910 2173 log.cc:824] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:43.108942 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: No bootstrap required, opened a new log
I20260620 08:54:43.109313 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.097051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.0.98.62:32791)
0620 08:54:43.097446 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.097461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.097494 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:54:43.098119 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.098134 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.098455 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.098718 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.099565 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.100589 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.101716 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.102200 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.102332 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.105019 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.105024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.105036 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.105062 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.106639 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.107112 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.107117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.107121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.107226 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.107484 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.107487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.107488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.107928 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.108073 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.108327 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.108592 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
I20260620 08:54:43.111855 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e94064ca2cc424c90ef6f2657ce5469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32791 } }
I20260620 08:54:43.112056 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.112102 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e94064ca2cc424c90ef6f2657ce5469, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.112488 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e94064ca2cc424c90ef6f2657ce5469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32791 } }
I20260620 08:54:43.112603 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:43.112645 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:43.112732 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:43.113571 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e94064ca2cc424c90ef6f2657ce5469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32791 } }
I20260620 08:54:43.113993 2173 leader_election.cc:302] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e94064ca2cc424c90ef6f2657ce5469; no voters:
I20260620 08:54:43.114282 2173 leader_election.cc:288] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:43.114544 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:43.114763 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [term 1 LEADER]: Becoming Leader. State: Replica: 0e94064ca2cc424c90ef6f2657ce5469, State: Running, Role: LEADER
I20260620 08:54:43.115067 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e94064ca2cc424c90ef6f2657ce5469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32791 } }
I20260620 08:54:43.115592 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:43.116667 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e94064ca2cc424c90ef6f2657ce5469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e94064ca2cc424c90ef6f2657ce5469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32791 } } }
I20260620 08:54:43.116770 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:43.117069 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e94064ca2cc424c90ef6f2657ce5469. Latest consensus state: current_term: 1 leader_uuid: "0e94064ca2cc424c90ef6f2657ce5469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e94064ca2cc424c90ef6f2657ce5469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32791 } } }
I20260620 08:54:43.117159 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:43.117236 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:43.119886 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:43.124671 2187 catalog_manager.cc:1347] Generated new cluster ID: 0fe96f92128e4a16904902d4ecc30ce5
I20260620 08:54:43.124738 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:43.129257 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:43.130381 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:43.139938 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Generated new TSK 0
I20260620 08:54:43.140710 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:43.170463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32791
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:43.283334 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:43.283593 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:43.283663 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:43.287928 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:43.287987 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:43.288028 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:54:43.288062 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:54:43.288115 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:43.293382 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:43.294555 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:43.295668 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:43.305513 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.305524 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.305524 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:43.305935 2199 server_base.cc:1034] running on GCE node
I20260620 08:54:43.306285 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:43.306865 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:43.308039 2199 hybrid_clock.cc:648] HybridClock initialized: now 1781945683308012 us; error 38 us; skew 500 ppm
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:43.310720 2199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:43.311522 2199 webserver.cc:460] Webserver started at http://127.0.98.1:35217/ using document root <none> and password file <none>
I20260620 08:54:43.312067 2199 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:43.312140 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:43.312328 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:43.314250 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ae06d1b9c07485185beda9112176f6b"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "4ae5f546dc89b1c0ae3bb7fafb7f6d65"
server_key_iv: "a6ec487c0df3b7f579c705cd65153be9"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.314766 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ae06d1b9c07485185beda9112176f6b"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "4ae5f546dc89b1c0ae3bb7fafb7f6d65"
server_key_iv: "a6ec487c0df3b7f579c705cd65153be9"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.318459 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:54:43.321991 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.323585 2199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:54:43.323745 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "4ae06d1b9c07485185beda9112176f6b"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "4ae5f546dc89b1c0ae3bb7fafb7f6d65"
server_key_iv: "a6ec487c0df3b7f579c705cd65153be9"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.323877 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:43.339633 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:43.343086 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:43.343273 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:43.344663 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:43.344710 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.344772 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:43.344799 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.354887 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42881
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:43.355188 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42881 every 8 connection(s)
I20260620 08:54:43.356009 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:43.356987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 2199
I20260620 08:54:43.357097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:43.357339 392 external_mini_cluster.cc:1428] Setting key 60cfdf6cf6a39bea84119dd0d155474f
I20260620 08:54:43.359750 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.345629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47369 (local address 127.0.98.62:32791)
0620 08:54:43.345756 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.345759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.346774 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 08:54:43.347830 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.347834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.347897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.347954 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.349688 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.350542 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.351712 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.352033 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.352129 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.355947 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.355953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.355956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.355985 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.357654 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.358212 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.358215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.358216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.358297 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.358648 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.358652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.358654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.358899 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.358992 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.359438 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.359564 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:54:43.360496 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.346128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.98.1:47369)
0620 08:54:43.346601 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.346640 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.347573 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.348368 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.348377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.348839 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.349490 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.349510 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.350666 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.350670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.351526 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.351535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.352339 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.352344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.352529 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.353602 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.353625 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.355710 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.357797 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.357803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.357817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.358099 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.358409 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.358412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.358414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.358554 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.358991 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.358997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.359221 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.359382 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.359711 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260620 08:54:43.361688 2330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32791
I20260620 08:54:43.361872 2330 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:43.362448 2330 heartbeater.cc:511] Master 127.0.98.62:32791 requested a full tablet report, sending...
I20260620 08:54:43.364050 2127 ts_manager.cc:194] Registered new tserver with Master: 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881)
I20260620 08:54:43.365353 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47369
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:54:43.372763 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.364087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46725 (local address 127.0.98.62:32791)
0620 08:54:43.364278 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.364282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.364527 (+ 245us) server_negotiation.cc:407] Connection header received
0620 08:54:43.364634 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.364636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.364677 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.364753 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.365308 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.366015 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.368178 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.368424 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.368509 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.369694 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.369697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.369700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.369720 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.371281 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.371780 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.371783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.371784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.371852 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.372107 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.372109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.372110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.372257 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.372325 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.372565 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.372653 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 08:54:43.372951 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.364009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.98.1:46725)
0620 08:54:43.364433 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.364446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.364554 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.364782 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.364785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.365030 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.365150 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.365158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.366146 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.366149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.368038 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.368049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.368563 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.368566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.368648 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.369199 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.369210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.369542 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.371409 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.371413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.371415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.371696 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.371943 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.371945 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.371947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.372035 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.372337 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.372339 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.372428 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.372528 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.372660 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:43.386914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32791
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:43.495688 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:43.495936 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:43.495981 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:43.499899 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:43.499954 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:43.499992 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:54:43.500005 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:54:43.500034 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:43.505095 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:43.506000 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:43.506901 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:54:43.516000 2339 server_base.cc:1034] running on GCE node
W20260620 08:54:43.515947 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.515950 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.515914 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:43.516567 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:43.517129 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:43.518328 2339 hybrid_clock.cc:648] HybridClock initialized: now 1781945683518312 us; error 33 us; skew 500 ppm
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:43.520906 2339 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:43.521697 2339 webserver.cc:460] Webserver started at http://127.0.98.2:33461/ using document root <none> and password file <none>
I20260620 08:54:43.522215 2339 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:43.522276 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:43.522461 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:43.524394 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3f037dc24004201bd934e2ec88d1ead"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "a1572c5c6952d063292a9c1f1a16a31c"
server_key_iv: "e9dc1a5451e0bad9448b9eba2146b848"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.524888 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3f037dc24004201bd934e2ec88d1ead"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "a1572c5c6952d063292a9c1f1a16a31c"
server_key_iv: "e9dc1a5451e0bad9448b9eba2146b848"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.528395 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:43.530586 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.531791 2339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:43.531893 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "c3f037dc24004201bd934e2ec88d1ead"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "a1572c5c6952d063292a9c1f1a16a31c"
server_key_iv: "e9dc1a5451e0bad9448b9eba2146b848"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.531976 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:43.553292 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:43.556020 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:43.556181 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:43.557456 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:43.557502 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.557551 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:43.557580 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:43.567672 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36393
I20260620 08:54:43.567747 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36393 every 8 connection(s)
I20260620 08:54:43.568605 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:43.570834 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.558346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48835 (local address 127.0.98.62:32791)
0620 08:54:43.558479 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.558484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.559257 (+ 773us) server_negotiation.cc:407] Connection header received
0620 08:54:43.560319 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.560322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.560376 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.560452 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.561877 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.562752 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.564401 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.564633 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.564716 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.567662 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.567667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.567669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.567698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.569110 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.569630 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.569633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.569636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.569731 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.569974 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.569976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.569977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.570138 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.570206 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.570622 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.570718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:54:43.571537 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.558670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.98.2:48835)
0620 08:54:43.559101 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.559148 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.560106 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.560646 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.560656 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.561071 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.561714 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.561733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.562869 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.562872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.564275 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.564288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.564786 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.564790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.564980 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.565662 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.565683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.567475 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.569221 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.569226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.569238 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.569537 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.569790 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.569792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.569794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.569912 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.570208 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.570213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.570422 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.570565 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.570789 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260620 08:54:43.572510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 2339
I20260620 08:54:43.572602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:43.572762 2470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32791
I20260620 08:54:43.572896 2470 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:43.572913 392 external_mini_cluster.cc:1428] Setting key 8b7d06764378fa490300b635303c8936
I20260620 08:54:43.573415 2470 heartbeater.cc:511] Master 127.0.98.62:32791 requested a full tablet report, sending...
I20260620 08:54:43.574332 2126 ts_manager.cc:194] Registered new tserver with Master: c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393)
I20260620 08:54:43.575011 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48835
I20260620 08:54:43.581040 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.574715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45415 (local address 127.0.98.62:32791)
0620 08:54:43.574822 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.574826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.575068 (+ 242us) server_negotiation.cc:407] Connection header received
0620 08:54:43.575216 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.575219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.575269 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.575334 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.575850 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.576339 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.577096 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.577304 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.577385 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.578320 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.578324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.578326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.578349 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.579568 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.579958 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.579961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.579963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.580032 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.580264 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.580267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.580268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.580455 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.580539 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.580800 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.580901 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:54:43.581049 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.574769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.98.2:45415)
0620 08:54:43.574994 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.575005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.575131 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.575435 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.575438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.575647 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.575750 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.575756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.576451 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.576454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.576997 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.577004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.577397 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.577401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.577490 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.577906 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.577916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.578199 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.579682 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.579685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.579687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.579872 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.580110 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.580112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.580114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.580189 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.580541 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.580544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.580651 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.580744 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.580870 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:43.600776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32791
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:43.705987 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:43.706216 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:43.706261 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:43.710152 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:43.710201 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:43.710237 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:54:43.710253 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:54:43.710307 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:43.715395 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:43.716360 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:43.717329 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:43.726490 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:43.726549 2479 server_base.cc:1034] running on GCE node
W20260620 08:54:43.726496 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:43.726496 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:43.727057 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:43.727787 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:43.728981 2479 hybrid_clock.cc:648] HybridClock initialized: now 1781945683728964 us; error 30 us; skew 500 ppm
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:43.732187 2479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:43.733037 2479 webserver.cc:460] Webserver started at http://127.0.98.3:46573/ using document root <none> and password file <none>
I20260620 08:54:43.733568 2479 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:43.733629 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:43.733806 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:43.735893 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "382825d452d4472ca6876c0a8565d69a"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "022a21c9b21c1b5df8f7ccfa55775656"
server_key_iv: "f3ec96ecfc906b1b2948cbcdbf07025e"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.736644 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "382825d452d4472ca6876c0a8565d69a"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "022a21c9b21c1b5df8f7ccfa55775656"
server_key_iv: "f3ec96ecfc906b1b2948cbcdbf07025e"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.740582 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 08:54:43.742955 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.744266 2479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:43.744374 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "382825d452d4472ca6876c0a8565d69a"
format_stamp: "Formatted at 2026-06-20 08:54:43 on dist-test-slave-fb1v"
server_key: "022a21c9b21c1b5df8f7ccfa55775656"
server_key_iv: "f3ec96ecfc906b1b2948cbcdbf07025e"
server_key_version: "encryptionkey@0"
I20260620 08:54:43.744473 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:43.761478 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:43.764250 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:43.764411 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:43.765738 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:43.765784 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:43.765909 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:43.765944 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:43.775514 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36565
I20260620 08:54:43.775553 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36565 every 8 connection(s)
I20260620 08:54:43.776507 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:43.779091 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.766546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35887 (local address 127.0.98.62:32791)
0620 08:54:43.766652 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.766655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.767694 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 08:54:43.768661 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.768664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.768718 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.768817 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.770127 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.770929 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.772600 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.772911 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.773013 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.775575 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.775579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.775582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.775637 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.777096 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.777644 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.777647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.777648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.777728 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.778060 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.778064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.778066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.778247 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.778344 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.778809 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.778940 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:54:43.779805 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.767100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.98.3:35887)
0620 08:54:43.767546 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.767584 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.768458 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.768953 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.768962 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.769410 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.769994 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.770012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.771092 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.771096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.772478 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.772490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.773060 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.773065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.773259 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.773891 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.773910 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.775384 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.777217 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.777222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.777234 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.777536 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.777812 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.777814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.777815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.777946 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.778351 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.778365 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.778589 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.778749 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.779031 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":134,"threads_started":1}
I20260620 08:54:43.780913 2610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32791
I20260620 08:54:43.781042 2610 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:43.781499 2610 heartbeater.cc:511] Master 127.0.98.62:32791 requested a full tablet report, sending...
I20260620 08:54:43.782418 2127 ts_manager.cc:194] Registered new tserver with Master: 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:43.783015 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35887
I20260620 08:54:43.786264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 2479
I20260620 08:54:43.786365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:43.786711 392 external_mini_cluster.cc:1428] Setting key 28000be398363177d2dde6d07f5d7c7c
I20260620 08:54:43.788621 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:54:43.789436 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.782713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53765 (local address 127.0.98.62:32791)
0620 08:54:43.783262 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.783265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.783275 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:54:43.783299 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.783301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.783329 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.783389 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.783814 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.784531 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.785246 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.785447 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.785520 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.786545 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.786549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.786551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.786570 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.787993 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.788421 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.788424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.788426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.788509 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.788779 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.788781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.788782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.788948 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.789018 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.789254 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.789332 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495}
I20260620 08:54:43.789521 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.782653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.98.3:53765)
0620 08:54:43.782873 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.782885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.782951 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.783423 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.783427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.783587 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.783715 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.783721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.784626 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.784628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.785159 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.785165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.785526 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.785529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.785587 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.785970 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.785981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.786364 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.788086 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.788090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.788092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.788323 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.788598 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.788601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.788603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.788703 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.789024 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.789027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.789119 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.789222 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.789351 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:43.812180 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.804604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.98.62:32791)
0620 08:54:43.804720 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.804723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.804735 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:54:43.804771 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.804773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.804836 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.804910 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.805667 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.806202 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.807054 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.807265 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.807341 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.809487 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.809490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.809492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.809513 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.810692 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.811117 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.811120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.811121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.811204 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.811443 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.811445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.811446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.811639 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.811753 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.811972 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.812063 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:54:43.814924 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:54:43.818025 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:43.831794 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.825932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.0.98.3:36565)
0620 08:54:43.826940 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.826951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.826969 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:54:43.827038 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.827042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.827221 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.827417 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:43.827870 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.828657 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.830233 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.831053 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.831164 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.831336 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.831528 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.831646 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":100,"threads_started":1}
I20260620 08:54:43.831799 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.825629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36393 (local address 127.0.0.1:59698)
0620 08:54:43.826416 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.826510 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.826639 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.827362 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.827368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.827392 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:43.827494 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.827499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.829189 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.829192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.830279 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.830288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.831280 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.831284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.831369 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.831391 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.831441 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.831576 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":36,"spinlock_wait_cycles":4736,"thread_start_us":69,"threads_started":1}
I20260620 08:54:43.832029 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.826351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.98.1:42881)
0620 08:54:43.826679 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.826694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.826968 (+ 274us) server_negotiation.cc:407] Connection header received
0620 08:54:43.827152 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.827157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.827316 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.827417 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:43.827905 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.829032 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.830440 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.831100 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.831232 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.831394 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.831769 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.831882 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":72,"threads_started":1}
I20260620 08:54:43.832111 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.826451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.0.98.2:36393)
0620 08:54:43.826878 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.826890 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.826913 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:54:43.826987 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.826993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.827194 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.827743 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:43.827890 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.829063 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.830413 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.831149 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.831708 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.831855 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.831906 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.831986 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":110,"threads_started":1}
I20260620 08:54:43.832381 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.825934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.0.1:34236)
0620 08:54:43.826883 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.826895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.827033 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.827584 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.827587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.827637 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:43.827773 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.827779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.829148 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.829151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.830347 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.830354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.831457 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.831461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.831545 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.831569 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.832160 (+ 591us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.832236 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":134,"spinlock_wait_cycles":384,"thread_start_us":71,"threads_started":1}
I20260620 08:54:43.832526 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.825737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36565 (local address 127.0.0.1:59520)
0620 08:54:43.826476 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.826510 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.826639 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.827547 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.827550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.827568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:43.827731 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.827738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.828752 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.828755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.830115 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.830125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.831206 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.831211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.831321 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.831361 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.831476 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.831588 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":171,"spinlock_wait_cycles":3328,"thread_start_us":100,"threads_started":1}
I20260620 08:54:43.834897 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7d57f2986c4ed2856b56c9850e55e5 (DEFAULT_TABLE table=test-table [id=d83162d736e84bd79a4ad703b7646771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:43.834879 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7d57f2986c4ed2856b56c9850e55e5 (DEFAULT_TABLE table=test-table [id=d83162d736e84bd79a4ad703b7646771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:43.835320 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7d57f2986c4ed2856b56c9850e55e5 (DEFAULT_TABLE table=test-table [id=d83162d736e84bd79a4ad703b7646771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:43.835925 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7d57f2986c4ed2856b56c9850e55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.835933 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7d57f2986c4ed2856b56c9850e55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.836228 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7d57f2986c4ed2856b56c9850e55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.843037 2633 tablet_bootstrap.cc:492] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Bootstrap starting.
I20260620 08:54:43.844002 2635 tablet_bootstrap.cc:492] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Bootstrap starting.
I20260620 08:54:43.845007 2633 tablet_bootstrap.cc:654] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.845659 2635 tablet_bootstrap.cc:654] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.845664 2633 log.cc:824] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:43.846302 2635 log.cc:824] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:43.846848 2634 tablet_bootstrap.cc:492] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Bootstrap starting.
I20260620 08:54:43.847344 2633 tablet_bootstrap.cc:492] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: No bootstrap required, opened a new log
I20260620 08:54:43.847517 2633 ts_tablet_manager.cc:1397] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:54:43.848227 2635 tablet_bootstrap.cc:492] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: No bootstrap required, opened a new log
I20260620 08:54:43.848407 2635 ts_tablet_manager.cc:1397] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:54:43.848881 2634 tablet_bootstrap.cc:654] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.849689 2634 log.cc:824] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:43.851174 2633 raft_consensus.cc:348] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.851377 2633 raft_consensus.cc:374] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.851425 2633 raft_consensus.cc:729] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.851538 2634 tablet_bootstrap.cc:492] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: No bootstrap required, opened a new log
I20260620 08:54:43.851743 2634 ts_tablet_manager.cc:1397] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:54:43.851881 2633 consensus_queue.cc:260] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.852056 2635 raft_consensus.cc:348] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.852298 2635 raft_consensus.cc:374] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.852358 2635 raft_consensus.cc:729] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382825d452d4472ca6876c0a8565d69a, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.852728 2633 ts_tablet_manager.cc:1428] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:54:43.852766 2635 consensus_queue.cc:260] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.853061 2330 heartbeater.cc:503] Master 127.0.98.62:32791 was elected leader, sending a full tablet report...
I20260620 08:54:43.853443 2610 heartbeater.cc:503] Master 127.0.98.62:32791 was elected leader, sending a full tablet report...
I20260620 08:54:43.853696 2635 ts_tablet_manager.cc:1428] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:54:43.855365 2634 raft_consensus.cc:348] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.855576 2634 raft_consensus.cc:374] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.855656 2634 raft_consensus.cc:729] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.856055 2634 consensus_queue.cc:260] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.856662 2470 heartbeater.cc:503] Master 127.0.98.62:32791 was elected leader, sending a full tablet report...
I20260620 08:54:43.856964 2634 ts_tablet_manager.cc:1428] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 08:54:43.857813 2331 tablet.cc:2369] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:43.889075 2639 raft_consensus.cc:482] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:43.889264 2639 raft_consensus.cc:504] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.890340 2639 leader_election.cc:288] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:43.894112 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.890716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42753 (local address 127.0.98.2:36393)
0620 08:54:43.890852 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.890856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.890992 (+ 136us) server_negotiation.cc:407] Connection header received
0620 08:54:43.891130 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.891133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.891175 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.891245 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:43.891477 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.892047 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.893147 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.893543 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.893635 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.893697 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.893923 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.893995 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:54:43.894115 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.890735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34003 (local address 127.0.98.3:36565)
0620 08:54:43.890850 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.890854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.891093 (+ 239us) server_negotiation.cc:407] Connection header received
0620 08:54:43.891209 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.891211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.891257 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.891320 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:43.891548 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.892085 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.893154 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.893547 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.893637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.893695 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.893931 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.893998 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:54:43.894129 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.890589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36393 (local address 127.0.98.1:42753)
0620 08:54:43.890906 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.890920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.891027 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.891257 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.891259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.891274 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:43.891383 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.891388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.892155 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.892159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.893035 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.893044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.893656 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.893659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.893813 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.893837 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.893898 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.893987 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":57,"thread_start_us":66,"threads_started":1}
I20260620 08:54:43.894140 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.890589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36565 (local address 127.0.98.1:34003)
0620 08:54:43.891006 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.891017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.891115 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.891335 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.891338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.891352 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:43.891456 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.891462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.892184 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.892187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.893037 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.893046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.893656 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.893659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.893794 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.893839 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.893901 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.894000 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":286,"thread_start_us":48,"threads_started":1}
I20260620 08:54:43.894680 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7d57f2986c4ed2856b56c9850e55e5" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f037dc24004201bd934e2ec88d1ead" is_pre_election: true
I20260620 08:54:43.894827 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7d57f2986c4ed2856b56c9850e55e5" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a" is_pre_election: true
I20260620 08:54:43.894972 2424 raft_consensus.cc:2454] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 0.
I20260620 08:54:43.895097 2564 raft_consensus.cc:2454] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 0.
I20260620 08:54:43.895399 2218 leader_election.cc:302] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae06d1b9c07485185beda9112176f6b, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:43.895654 2639 raft_consensus.cc:2781] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:43.895735 2639 raft_consensus.cc:482] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:43.895884 2639 raft_consensus.cc:3037] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:43.896823 2639 raft_consensus.cc:504] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.897119 2639 leader_election.cc:288] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 election: Requested vote from peers c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:43.897423 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7d57f2986c4ed2856b56c9850e55e5" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a"
I20260620 08:54:43.897414 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7d57f2986c4ed2856b56c9850e55e5" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f037dc24004201bd934e2ec88d1ead"
I20260620 08:54:43.897540 2564 raft_consensus.cc:3037] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:43.897605 2424 raft_consensus.cc:3037] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:43.898746 2424 raft_consensus.cc:2454] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 1.
I20260620 08:54:43.898746 2564 raft_consensus.cc:2454] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 1.
I20260620 08:54:43.899024 2218 leader_election.cc:302] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae06d1b9c07485185beda9112176f6b, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:43.899191 2639 raft_consensus.cc:2781] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:43.899396 2639 raft_consensus.cc:686] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 1 LEADER]: Becoming Leader. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Running, Role: LEADER
I20260620 08:54:43.899717 2639 consensus_queue.cc:237] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } }
I20260620 08:54:43.902748 2127 catalog_manager.cc:5495] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae06d1b9c07485185beda9112176f6b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } health_report { overall_health: HEALTHY } } }
Jun 20 08:54:43 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:43.957096 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.949327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.0.98.62:32791)
0620 08:54:43.949467 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:43.949471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:43.949640 (+ 169us) server_negotiation.cc:407] Connection header received
0620 08:54:43.949737 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:43.949739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:43.949783 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:43.949861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:43.950514 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.951030 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.951855 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.952059 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.952135 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.954220 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:43.954223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:43.954225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:43.954244 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:43.955481 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.955957 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.955959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.955960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.956026 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:43.956316 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:43.956319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:43.956320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:43.956488 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:43.956561 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:43.956830 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 08:54:43.956965 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":71}
I20260620 08:54:43.957098 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:43.949247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32791 (local address 127.0.0.1:44014)
0620 08:54:43.949529 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:43.949544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:43.949649 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:43.949917 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:43.949919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:43.950291 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:43.950381 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.950387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.951130 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.951132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.951728 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:43.951743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:43.952140 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:43.952142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:43.952197 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:43.952692 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:43.952703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:43.954044 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:43.955587 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.955596 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.955600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.955878 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.956126 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:43.956128 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:43.956129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:43.956215 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:43.956582 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:43.956587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:43.956688 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 08:54:43.956791 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:54:43.956913 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260620 08:54:43.961212 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44014:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:54:43.961916 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:43.966576 2263 tablet_service.cc:1467] Processing CreateTablet for tablet a68883b6b2d04b5f91f3aaa66fbdd6ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d210d7276b040abbad19eaea3b63cd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:54:43.967242 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68883b6b2d04b5f91f3aaa66fbdd6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.967105 2404 tablet_service.cc:1467] Processing CreateTablet for tablet a68883b6b2d04b5f91f3aaa66fbdd6ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d210d7276b040abbad19eaea3b63cd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:54:43.967060 2544 tablet_service.cc:1467] Processing CreateTablet for tablet a68883b6b2d04b5f91f3aaa66fbdd6ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d210d7276b040abbad19eaea3b63cd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:54:43.967715 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68883b6b2d04b5f91f3aaa66fbdd6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.967715 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68883b6b2d04b5f91f3aaa66fbdd6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:43.969767 2633 tablet_bootstrap.cc:492] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: Bootstrap starting.
I20260620 08:54:43.970486 2635 tablet_bootstrap.cc:492] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: Bootstrap starting.
I20260620 08:54:43.970695 2634 tablet_bootstrap.cc:492] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: Bootstrap starting.
I20260620 08:54:43.971092 2633 tablet_bootstrap.cc:654] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.971390 2635 tablet_bootstrap.cc:654] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.971443 2634 tablet_bootstrap.cc:654] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:43.972322 2633 tablet_bootstrap.cc:492] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: No bootstrap required, opened a new log
I20260620 08:54:43.972409 2633 ts_tablet_manager.cc:1397] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 08:54:43.972577 2634 tablet_bootstrap.cc:492] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: No bootstrap required, opened a new log
I20260620 08:54:43.972651 2634 ts_tablet_manager.cc:1397] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:43.972885 2635 tablet_bootstrap.cc:492] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: No bootstrap required, opened a new log
I20260620 08:54:43.972884 2633 raft_consensus.cc:348] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:43.972954 2635 ts_tablet_manager.cc:1397] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:54:43.972983 2633 raft_consensus.cc:374] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.973006 2633 raft_consensus.cc:729] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.973075 2634 raft_consensus.cc:348] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:43.973105 2633 consensus_queue.cc:260] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:43.973170 2634 raft_consensus.cc:374] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.973243 2634 raft_consensus.cc:729] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.973333 2639 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:43.973455 2639 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:43.973541 2635 raft_consensus.cc:348] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:43.973531 2634 consensus_queue.cc:260] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:43.973631 2635 raft_consensus.cc:374] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:43.973660 2635 raft_consensus.cc:729] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382825d452d4472ca6876c0a8565d69a, State: Initialized, Role: FOLLOWER
I20260620 08:54:43.973769 2635 consensus_queue.cc:260] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:43.973810 2634 ts_tablet_manager.cc:1428] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:43.973341 2633 ts_tablet_manager.cc:1428] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:54:43.974046 2640 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:43.974164 2640 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:43.974349 2635 ts_tablet_manager.cc:1428] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:43.973815 2641 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:43.974740 2641 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 08:54:44.027437 2611 tablet.cc:2369] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:54:44.069737 2471 tablet.cc:2369] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:44.280426 2639 raft_consensus.cc:482] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:44.280596 2639 raft_consensus.cc:504] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.281060 2639 leader_election.cc:288] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:44.281594 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68883b6b2d04b5f91f3aaa66fbdd6ac" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a" is_pre_election: true
I20260620 08:54:44.281629 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68883b6b2d04b5f91f3aaa66fbdd6ac" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f037dc24004201bd934e2ec88d1ead" is_pre_election: true
I20260620 08:54:44.281759 2564 raft_consensus.cc:2454] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 0.
I20260620 08:54:44.281759 2424 raft_consensus.cc:2454] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 0.
I20260620 08:54:44.282097 2216 leader_election.cc:302] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382825d452d4472ca6876c0a8565d69a, 4ae06d1b9c07485185beda9112176f6b; no voters:
I20260620 08:54:44.282271 2639 raft_consensus.cc:2781] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:44.282330 2639 raft_consensus.cc:482] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:44.282375 2639 raft_consensus.cc:3037] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:44.283380 2639 raft_consensus.cc:504] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.283727 2639 leader_election.cc:288] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 election: Requested vote from peers c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:44.284108 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68883b6b2d04b5f91f3aaa66fbdd6ac" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a"
I20260620 08:54:44.284080 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68883b6b2d04b5f91f3aaa66fbdd6ac" candidate_uuid: "4ae06d1b9c07485185beda9112176f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f037dc24004201bd934e2ec88d1ead"
I20260620 08:54:44.284226 2564 raft_consensus.cc:3037] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:44.284235 2424 raft_consensus.cc:3037] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:44.285288 2424 raft_consensus.cc:2454] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 1.
I20260620 08:54:44.285288 2564 raft_consensus.cc:2454] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae06d1b9c07485185beda9112176f6b in term 1.
I20260620 08:54:44.285586 2216 leader_election.cc:302] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382825d452d4472ca6876c0a8565d69a, 4ae06d1b9c07485185beda9112176f6b; no voters:
I20260620 08:54:44.285745 2639 raft_consensus.cc:2781] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:44.285837 2639 raft_consensus.cc:686] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [term 1 LEADER]: Becoming Leader. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Running, Role: LEADER
I20260620 08:54:44.285970 2639 consensus_queue.cc:237] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.286644 2644 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: TxnStatusTablet state changed. Reason: New leader 4ae06d1b9c07485185beda9112176f6b. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.286747 2644 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:54:44.287016 2656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:54:44.287230 2656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:54:44.287819 2126 catalog_manager.cc:5495] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae06d1b9c07485185beda9112176f6b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } health_report { overall_health: UNKNOWN } } }
I20260620 08:54:44.343731 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.340268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.0.98.1:42881)
0620 08:54:44.340523 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.340527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.340540 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:44.340605 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.340607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.340657 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.340743 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:44.340973 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.341513 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.342344 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.342581 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.342665 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.342934 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.343323 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.343421 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.343496 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.343548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260620 08:54:44.343716 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.340176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.0.1:34252)
0620 08:54:44.340313 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:44.340329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:44.340426 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:44.340766 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:44.340768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:44.340776 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:44.340877 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.340883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.341622 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.341626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.342235 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.342243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.342675 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.342678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.342766 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.342808 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.343410 (+ 602us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.343423 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 08:54:44.343485 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 08:54:44.343567 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 08:54:44.348651 2564 raft_consensus.cc:1261] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Refusing update from remote peer 4ae06d1b9c07485185beda9112176f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:44.348860 2424 raft_consensus.cc:1261] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Refusing update from remote peer 4ae06d1b9c07485185beda9112176f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:44.349313 2639 consensus_queue.cc:1035] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:44.349452 2644 consensus_queue.cc:1035] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:44.353384 2640 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: TxnStatusTablet state changed. Reason: New leader 4ae06d1b9c07485185beda9112176f6b. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.353502 2640 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:44.354043 2641 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: TxnStatusTablet state changed. Reason: New leader 4ae06d1b9c07485185beda9112176f6b. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.354156 2641 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:44.355763 2639 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.355888 2639 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:54:44.358955 2640 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.359071 2640 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 382825d452d4472ca6876c0a8565d69a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:44.360683 2661 mvcc.cc:205] Tried to move back new op lower bound from 7298849523085791232 to 7298849522836582400. Current Snapshot: MvccSnapshot[applied={T|T < 7298849523085791232}]
I20260620 08:54:44.366451 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.362377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.0.98.1:42881)
0620 08:54:44.362634 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.362639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.362793 (+ 154us) server_negotiation.cc:407] Connection header received
0620 08:54:44.362913 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.362916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.362958 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.363027 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:44.363251 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.364066 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.365185 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.365516 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.365624 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.365976 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.366050 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.366088 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.366271 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.366333 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260620 08:54:44.372295 2644 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.372418 2644 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P 4ae06d1b9c07485185beda9112176f6b: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:54:44.374044 2641 tablet_replica.cc:439] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae06d1b9c07485185beda9112176f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } } }
I20260620 08:54:44.374162 2641 tablet_replica.cc:442] T a68883b6b2d04b5f91f3aaa66fbdd6ac P c3f037dc24004201bd934e2ec88d1ead: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:54:44.374979 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298849523085791232 to 7298849522836582400. Current Snapshot: MvccSnapshot[applied={T|T < 7298849523085791232}]
I20260620 08:54:44.382812 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.373580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.98.1:47703)
0620 08:54:44.379314 (+ 5734us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:44.379337 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:44.379441 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:44.379817 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:44.379821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:44.379827 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:44.379930 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.379936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.380872 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.380875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.381567 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.381576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.382087 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.382090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.382183 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.382231 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.382569 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.382575 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:54:44.382628 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:54:44.382689 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5642,"thread_start_us":92,"threads_started":1}
I20260620 08:54:44.382822 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.373931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47703 (local address 127.0.98.1:42881)
0620 08:54:44.374116 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.374120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.379444 (+ 5324us) server_negotiation.cc:407] Connection header received
0620 08:54:44.379572 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.379576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.379655 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.379732 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:44.380025 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.380759 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.381673 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.381980 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.382087 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.382394 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.382482 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.382546 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.382651 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.382696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 08:54:44.388880 2564 raft_consensus.cc:1261] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Refusing update from remote peer 4ae06d1b9c07485185beda9112176f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:44.389212 2424 raft_consensus.cc:1261] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Refusing update from remote peer 4ae06d1b9c07485185beda9112176f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:44.389238 2644 consensus_queue.cc:1035] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:44.389556 2644 consensus_queue.cc:1035] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:44.394116 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298849523253743616 to 7298849521255731200. Current Snapshot: MvccSnapshot[applied={T|T < 7298849523253743616}]
I20260620 08:54:44.427439 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:44.427639 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:44.429890 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Sending DeleteTablet for 3 replicas of tablet 1a7d57f2986c4ed2856b56c9850e55e5
I20260620 08:54:44.430671 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7d57f2986c4ed2856b56c9850e55e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59698
I20260620 08:54:44.430667 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7d57f2986c4ed2856b56c9850e55e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59520
I20260620 08:54:44.431373 2679 tablet_replica.cc:330] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: stopping tablet replica
I20260620 08:54:44.431336 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7d57f2986c4ed2856b56c9850e55e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34236
I20260620 08:54:44.431372 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:54:44.431671 2678 tablet_replica.cc:330] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: stopping tablet replica
W20260620 08:54:44.431804 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:44.431883 2680 tablet_replica.cc:330] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: stopping tablet replica
I20260620 08:54:44.432173 2680 raft_consensus.cc:2229] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:44.432170 2679 raft_consensus.cc:2229] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:44.432410 2678 raft_consensus.cc:2229] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:44.432564 2680 raft_consensus.cc:2258] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:44.432765 2678 raft_consensus.cc:2258] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:44.432893 2679 raft_consensus.cc:2258] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:44.433918 2680 ts_tablet_manager.cc:1905] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:44.435024 2679 ts_tablet_manager.cc:1905] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:44.435380 2678 ts_tablet_manager.cc:1905] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:44.437199 2404 tablet_service.cc:1467] Processing CreateTablet for tablet d579ab8a378542039a5c3521747e7291 (DEFAULT_TABLE table=test-table [id=2c9738dd926e4367bb7fd313057c768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:44.437288 2544 tablet_service.cc:1467] Processing CreateTablet for tablet d579ab8a378542039a5c3521747e7291 (DEFAULT_TABLE table=test-table [id=2c9738dd926e4367bb7fd313057c768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:44.437592 2263 tablet_service.cc:1467] Processing CreateTablet for tablet d579ab8a378542039a5c3521747e7291 (DEFAULT_TABLE table=test-table [id=2c9738dd926e4367bb7fd313057c768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:44.437769 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d579ab8a378542039a5c3521747e7291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:44.437846 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d579ab8a378542039a5c3521747e7291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:44.439733 2633 tablet_bootstrap.cc:492] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Bootstrap starting.
I20260620 08:54:44.440116 2680 ts_tablet_manager.cc:1918] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:44.440191 2680 log.cc:1196] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/1a7d57f2986c4ed2856b56c9850e55e5
I20260620 08:54:44.440410 2680 ts_tablet_manager.cc:1939] T 1a7d57f2986c4ed2856b56c9850e55e5 P 4ae06d1b9c07485185beda9112176f6b: Deleting consensus metadata
I20260620 08:54:44.440541 2633 tablet_bootstrap.cc:654] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:44.441543 2113 catalog_manager.cc:4841] TS 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881): tablet 1a7d57f2986c4ed2856b56c9850e55e5 (table test-table [id=d83162d736e84bd79a4ad703b7646771]) successfully deleted
I20260620 08:54:44.441715 2635 tablet_bootstrap.cc:492] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Bootstrap starting.
I20260620 08:54:44.442082 2633 tablet_bootstrap.cc:492] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: No bootstrap required, opened a new log
I20260620 08:54:44.442165 2633 ts_tablet_manager.cc:1397] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:54:44.442538 2635 tablet_bootstrap.cc:654] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:44.442597 2633 raft_consensus.cc:348] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.442700 2633 raft_consensus.cc:374] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:44.442728 2633 raft_consensus.cc:729] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Initialized, Role: FOLLOWER
I20260620 08:54:44.442831 2633 consensus_queue.cc:260] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.443092 2633 ts_tablet_manager.cc:1428] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:54:44.447019 2678 ts_tablet_manager.cc:1918] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:44.447118 2678 log.cc:1196] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/1a7d57f2986c4ed2856b56c9850e55e5
I20260620 08:54:44.447391 2678 ts_tablet_manager.cc:1939] T 1a7d57f2986c4ed2856b56c9850e55e5 P 382825d452d4472ca6876c0a8565d69a: Deleting consensus metadata
I20260620 08:54:44.447739 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d579ab8a378542039a5c3521747e7291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:44.448310 2635 tablet_bootstrap.cc:492] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: No bootstrap required, opened a new log
I20260620 08:54:44.448385 2635 ts_tablet_manager.cc:1397] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 08:54:44.448804 2635 raft_consensus.cc:348] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.448905 2635 raft_consensus.cc:374] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:44.448932 2635 raft_consensus.cc:729] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382825d452d4472ca6876c0a8565d69a, State: Initialized, Role: FOLLOWER
I20260620 08:54:44.449034 2679 ts_tablet_manager.cc:1918] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:44.449033 2635 consensus_queue.cc:260] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.449115 2679 log.cc:1196] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/1a7d57f2986c4ed2856b56c9850e55e5
I20260620 08:54:44.449275 2635 ts_tablet_manager.cc:1428] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:44.449359 2679 ts_tablet_manager.cc:1939] T 1a7d57f2986c4ed2856b56c9850e55e5 P c3f037dc24004201bd934e2ec88d1ead: Deleting consensus metadata
I20260620 08:54:44.449920 2634 tablet_bootstrap.cc:492] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Bootstrap starting.
I20260620 08:54:44.450525 2114 catalog_manager.cc:4841] TS c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393): tablet 1a7d57f2986c4ed2856b56c9850e55e5 (table test-table [id=d83162d736e84bd79a4ad703b7646771]) successfully deleted
I20260620 08:54:44.450764 2634 tablet_bootstrap.cc:654] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:44.451320 2112 catalog_manager.cc:4841] TS 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565): tablet 1a7d57f2986c4ed2856b56c9850e55e5 (table test-table [id=d83162d736e84bd79a4ad703b7646771]) successfully deleted
I20260620 08:54:44.452404 2634 tablet_bootstrap.cc:492] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: No bootstrap required, opened a new log
I20260620 08:54:44.452479 2634 ts_tablet_manager.cc:1397] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 08:54:44.452839 2634 raft_consensus.cc:348] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.452926 2634 raft_consensus.cc:374] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:44.452955 2634 raft_consensus.cc:729] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Initialized, Role: FOLLOWER
I20260620 08:54:44.453068 2634 consensus_queue.cc:260] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.453312 2634 ts_tablet_manager.cc:1428] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:54:44.496179 2640 raft_consensus.cc:482] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:44.496317 2640 raft_consensus.cc:504] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.497417 2640 leader_election.cc:288] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881), c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393)
I20260620 08:54:44.501596 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.497650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36393 (local address 127.0.98.3:38701)
0620 08:54:44.498081 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:44.498092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:44.498156 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:44.498534 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:44.498537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:44.498555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:44.498674 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.498680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.499654 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.499657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.500492 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.500501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.501169 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.501172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.501268 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.501300 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 08:54:44.501368 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:54:44.501432 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":20,"thread_start_us":82,"threads_started":1}
I20260620 08:54:44.501603 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.497736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38701 (local address 127.0.98.2:36393)
0620 08:54:44.498300 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.498312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.498323 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:54:44.498371 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.498373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.498426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.498507 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:44.498786 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.499542 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.500609 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.501061 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.501164 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.501226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.501387 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.501460 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":90,"threads_started":1}
I20260620 08:54:44.502012 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579ab8a378542039a5c3521747e7291" candidate_uuid: "382825d452d4472ca6876c0a8565d69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f037dc24004201bd934e2ec88d1ead" is_pre_election: true
I20260620 08:54:44.502169 2424 raft_consensus.cc:2454] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382825d452d4472ca6876c0a8565d69a in term 0.
I20260620 08:54:44.502427 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.497937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52549 (local address 127.0.98.1:42881)
0620 08:54:44.498073 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.498077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.498092 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:54:44.498217 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.498220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.498266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.498355 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:44.498625 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.499387 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.500972 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.501486 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.501579 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.501638 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.502202 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.502302 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:54:44.502509 2498 leader_election.cc:302] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382825d452d4472ca6876c0a8565d69a, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:44.502737 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.497633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.98.3:52549)
0620 08:54:44.497991 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:44.498007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:44.498111 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:44.498378 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:44.498381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:44.498399 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:44.498528 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.498534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.499490 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.499494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.500851 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.500862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.501957 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.501960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.502060 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.502083 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:54:44.502549 (+ 466us) client_negotiation.cc:240] Negotiation successful
0620 08:54:44.502625 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":44,"thread_start_us":67,"threads_started":1}
I20260620 08:54:44.502771 2640 raft_consensus.cc:2781] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:44.502832 2640 raft_consensus.cc:482] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:44.502876 2640 raft_consensus.cc:3037] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:44.503198 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579ab8a378542039a5c3521747e7291" candidate_uuid: "382825d452d4472ca6876c0a8565d69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae06d1b9c07485185beda9112176f6b" is_pre_election: true
I20260620 08:54:44.503425 2277 raft_consensus.cc:2454] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382825d452d4472ca6876c0a8565d69a in term 0.
I20260620 08:54:44.504083 2640 raft_consensus.cc:504] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.504407 2640 leader_election.cc:288] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [CANDIDATE]: Term 1 election: Requested vote from peers 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881), c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393)
I20260620 08:54:44.504678 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579ab8a378542039a5c3521747e7291" candidate_uuid: "382825d452d4472ca6876c0a8565d69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f037dc24004201bd934e2ec88d1ead"
I20260620 08:54:44.504678 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d579ab8a378542039a5c3521747e7291" candidate_uuid: "382825d452d4472ca6876c0a8565d69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae06d1b9c07485185beda9112176f6b"
I20260620 08:54:44.504779 2277 raft_consensus.cc:3037] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:44.504781 2424 raft_consensus.cc:3037] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:44.505600 2424 raft_consensus.cc:2454] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382825d452d4472ca6876c0a8565d69a in term 1.
I20260620 08:54:44.505838 2277 raft_consensus.cc:2454] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382825d452d4472ca6876c0a8565d69a in term 1.
I20260620 08:54:44.505839 2498 leader_election.cc:302] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382825d452d4472ca6876c0a8565d69a, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:44.505999 2640 raft_consensus.cc:2781] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:44.506189 2640 raft_consensus.cc:686] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 1 LEADER]: Becoming Leader. State: Replica: 382825d452d4472ca6876c0a8565d69a, State: Running, Role: LEADER
I20260620 08:54:44.506420 2640 consensus_queue.cc:237] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:44.508883 2126 catalog_manager.cc:5495] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a reported cstate change: term changed from 0 to 1, leader changed from <none> to 382825d452d4472ca6876c0a8565d69a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "382825d452d4472ca6876c0a8565d69a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } health_report { overall_health: HEALTHY } } }
I20260620 08:54:44.534869 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.531157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.98.3:36565)
0620 08:54:44.531457 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.531465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.531519 (+ 54us) server_negotiation.cc:407] Connection header received
0620 08:54:44.531702 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.531705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.531769 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.531909 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:44.532120 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.532646 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.533434 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.533663 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.533748 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.534018 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.534476 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.534585 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.534652 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.534704 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":148,"threads_started":1}
I20260620 08:54:44.545956 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.541850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47505 (local address 127.0.98.1:42881)
0620 08:54:44.541983 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.541987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.542115 (+ 128us) server_negotiation.cc:407] Connection header received
0620 08:54:44.542246 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.542248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.542288 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.542341 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:44.542613 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.543255 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.544904 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.545114 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.545197 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.545516 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.545603 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.545710 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.545803 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.545849 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:54:44.546015 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.541736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.98.3:47505)
0620 08:54:44.542034 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:44.542049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:44.542147 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:44.542358 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:44.542361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:44.542367 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:44.542489 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.542494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.543350 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.543353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.544762 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.544773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.545211 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.545215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.545315 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.545363 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.545715 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.545720 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:54:44.545777 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:54:44.545854 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
I20260620 08:54:44.552568 2424 raft_consensus.cc:1261] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Refusing update from remote peer 382825d452d4472ca6876c0a8565d69a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:44.552654 2277 raft_consensus.cc:1261] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Refusing update from remote peer 382825d452d4472ca6876c0a8565d69a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:44.553274 2687 consensus_queue.cc:1035] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:44.553479 2640 consensus_queue.cc:1035] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:44.574033 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.569679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36565 (local address 127.0.98.1:57921)
0620 08:54:44.569919 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:44.569934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:44.570003 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:44.570536 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:44.570538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:44.570546 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:44.570658 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.570665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.571827 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.571831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.572646 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:44.572656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.573347 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.573350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:44.573428 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.573460 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.573774 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.573780 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:54:44.573850 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:54:44.573930 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 08:54:44.574452 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:44.570138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57921 (local address 127.0.98.3:36565)
0620 08:54:44.570256 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:44.570260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:44.570273 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:44.570311 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:44.570314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:44.570362 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:44.570425 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:44.571176 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.571731 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.572961 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:44.573247 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:44.573336 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:44.573584 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:44.573667 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:44.573737 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:44.574284 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 08:54:44.574339 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 08:54:44.575042 2661 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 08:54:46.140314 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:46.140465 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:46.142771 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Sending DeleteTablet for 3 replicas of tablet d579ab8a378542039a5c3521747e7291
I20260620 08:54:46.143285 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet d579ab8a378542039a5c3521747e7291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59698
I20260620 08:54:46.143275 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet d579ab8a378542039a5c3521747e7291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34236
I20260620 08:54:46.143625 2718 tablet_replica.cc:330] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: stopping tablet replica
I20260620 08:54:46.143819 2717 tablet_replica.cc:330] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: stopping tablet replica
I20260620 08:54:46.144001 2718 raft_consensus.cc:2229] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:46.144179 2717 raft_consensus.cc:2229] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:46.144307 2717 raft_consensus.cc:2258] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:46.144474 2718 raft_consensus.cc:2258] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:46.145036 2718 ts_tablet_manager.cc:1905] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:46.145217 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet d579ab8a378542039a5c3521747e7291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59520
I20260620 08:54:46.146199 2717 ts_tablet_manager.cc:1905] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:46.147732 2718 ts_tablet_manager.cc:1918] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:54:46.147832 2718 log.cc:1196] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/d579ab8a378542039a5c3521747e7291
I20260620 08:54:46.148087 2718 ts_tablet_manager.cc:1939] T d579ab8a378542039a5c3521747e7291 P 4ae06d1b9c07485185beda9112176f6b: Deleting consensus metadata
I20260620 08:54:46.148391 2717 ts_tablet_manager.cc:1918] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:54:46.148456 2717 log.cc:1196] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/d579ab8a378542039a5c3521747e7291
I20260620 08:54:46.148697 2717 ts_tablet_manager.cc:1939] T d579ab8a378542039a5c3521747e7291 P c3f037dc24004201bd934e2ec88d1ead: Deleting consensus metadata
I20260620 08:54:46.148782 2113 catalog_manager.cc:4841] TS 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881): tablet d579ab8a378542039a5c3521747e7291 (table test-table [id=2c9738dd926e4367bb7fd313057c768c]) successfully deleted
I20260620 08:54:46.149281 2720 tablet_replica.cc:330] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: stopping tablet replica
I20260620 08:54:46.149278 2114 catalog_manager.cc:4841] TS c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393): tablet d579ab8a378542039a5c3521747e7291 (table test-table [id=2c9738dd926e4367bb7fd313057c768c]) successfully deleted
I20260620 08:54:46.149400 2720 raft_consensus.cc:2229] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:46.149681 2720 raft_consensus.cc:2258] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:46.150149 2720 ts_tablet_manager.cc:1905] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:46.152077 2720 ts_tablet_manager.cc:1918] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:54:46.152139 2720 log.cc:1196] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/d579ab8a378542039a5c3521747e7291
I20260620 08:54:46.152345 2720 ts_tablet_manager.cc:1939] T d579ab8a378542039a5c3521747e7291 P 382825d452d4472ca6876c0a8565d69a: Deleting consensus metadata
I20260620 08:54:46.152861 2112 catalog_manager.cc:4841] TS 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565): tablet d579ab8a378542039a5c3521747e7291 (table test-table [id=2c9738dd926e4367bb7fd313057c768c]) successfully deleted
Jun 20 08:54:46 dist-test-slave-fb1v krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:54:46 dist-test-slave-fb1v krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:46.168572 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.160121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.0.98.62:32791)
0620 08:54:46.160368 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:46.160374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:46.160387 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:46.160541 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:46.160544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:46.160590 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:46.160681 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:46.161458 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.162116 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.162994 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.163239 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.163326 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.165706 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:46.165709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:46.165711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:46.165733 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:46.167134 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:46.167513 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:46.167516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:46.167517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:46.167595 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:46.167851 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:46.167853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:46.167854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:46.168019 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:46.168114 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:46.168288 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 08:54:46.168395 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260620 08:54:46.170861 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:54:46.171218 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:46.175416 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 6e19f3db2d74413b8d3620072a8c61f7 (DEFAULT_TABLE table=test-table [id=72f2e123fa2243a1a09af1443a97ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:46.175447 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 6e19f3db2d74413b8d3620072a8c61f7 (DEFAULT_TABLE table=test-table [id=72f2e123fa2243a1a09af1443a97ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:46.175455 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 6e19f3db2d74413b8d3620072a8c61f7 (DEFAULT_TABLE table=test-table [id=72f2e123fa2243a1a09af1443a97ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:46.175717 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e19f3db2d74413b8d3620072a8c61f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:46.175717 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e19f3db2d74413b8d3620072a8c61f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:46.175731 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e19f3db2d74413b8d3620072a8c61f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:46.177626 2728 tablet_bootstrap.cc:492] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Bootstrap starting.
I20260620 08:54:46.178011 2730 tablet_bootstrap.cc:492] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Bootstrap starting.
I20260620 08:54:46.178426 2729 tablet_bootstrap.cc:492] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Bootstrap starting.
I20260620 08:54:46.178519 2728 tablet_bootstrap.cc:654] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:46.178835 2730 tablet_bootstrap.cc:654] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:46.179261 2729 tablet_bootstrap.cc:654] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:46.179688 2728 tablet_bootstrap.cc:492] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: No bootstrap required, opened a new log
I20260620 08:54:46.179767 2728 ts_tablet_manager.cc:1397] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:46.180128 2728 raft_consensus.cc:348] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.180220 2728 raft_consensus.cc:374] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:46.180249 2728 raft_consensus.cc:729] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Initialized, Role: FOLLOWER
I20260620 08:54:46.180289 2729 tablet_bootstrap.cc:492] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: No bootstrap required, opened a new log
I20260620 08:54:46.180383 2729 ts_tablet_manager.cc:1397] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:46.180369 2728 consensus_queue.cc:260] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.180711 2728 ts_tablet_manager.cc:1428] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:46.180747 2729 raft_consensus.cc:348] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.180902 2729 raft_consensus.cc:374] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:46.180972 2729 raft_consensus.cc:729] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Initialized, Role: FOLLOWER
I20260620 08:54:46.181173 2729 consensus_queue.cc:260] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.181455 2730 tablet_bootstrap.cc:492] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: No bootstrap required, opened a new log
I20260620 08:54:46.181519 2729 ts_tablet_manager.cc:1428] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:46.181574 2730 ts_tablet_manager.cc:1397] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 08:54:46.182127 2730 raft_consensus.cc:348] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.182217 2730 raft_consensus.cc:374] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:46.182246 2730 raft_consensus.cc:729] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382825d452d4472ca6876c0a8565d69a, State: Initialized, Role: FOLLOWER
I20260620 08:54:46.182397 2730 consensus_queue.cc:260] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.183245 2730 ts_tablet_manager.cc:1428] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:46.303066 2734 raft_consensus.cc:482] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:46.303220 2734 raft_consensus.cc:504] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.304322 2734 leader_election.cc:288] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:46.308120 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.304700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55241 (local address 127.0.98.1:42881)
0620 08:54:46.304947 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:46.304954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:46.304972 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:54:46.305128 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:46.305130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:46.305168 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:46.305237 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:46.305472 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.306090 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.307164 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.307574 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.307700 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.307762 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:46.307915 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 08:54:46.307989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260620 08:54:46.308135 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.304662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54703 (local address 127.0.98.3:36565)
0620 08:54:46.304970 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:46.304978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:46.305062 (+ 84us) server_negotiation.cc:407] Connection header received
0620 08:54:46.305187 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:46.305190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:46.305232 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:46.305302 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:46.305553 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.306048 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.307154 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.307573 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.307698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.307762 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:46.307941 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 08:54:46.308010 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20260620 08:54:46.308158 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.304564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36565 (local address 127.0.98.2:54703)
0620 08:54:46.304966 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:46.304980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:46.305092 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:46.305320 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:46.305323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:46.305337 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:46.305455 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.305460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.306150 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.306154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.307033 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.307042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.307700 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.307703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.307794 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.307844 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 08:54:46.307915 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:54:46.307989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":75,"thread_start_us":115,"threads_started":1}
I20260620 08:54:46.308176 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.304522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.98.2:55241)
0620 08:54:46.304904 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:46.304918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:46.305024 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:46.305255 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:46.305257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:46.305271 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:46.305375 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.305380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.306214 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.306217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.307047 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.307056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.307702 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.307705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.307789 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.307828 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:54:46.307892 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:54:46.307965 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":19,"thread_start_us":138,"threads_started":1}
I20260620 08:54:46.308547 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e19f3db2d74413b8d3620072a8c61f7" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a" is_pre_election: true
I20260620 08:54:46.308699 2563 raft_consensus.cc:2454] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 0.
I20260620 08:54:46.308867 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e19f3db2d74413b8d3620072a8c61f7" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae06d1b9c07485185beda9112176f6b" is_pre_election: true
I20260620 08:54:46.309003 2277 raft_consensus.cc:2454] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 0.
I20260620 08:54:46.309089 2356 leader_election.cc:302] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382825d452d4472ca6876c0a8565d69a, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:46.309350 2734 raft_consensus.cc:2781] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:46.309422 2734 raft_consensus.cc:482] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:46.309464 2734 raft_consensus.cc:3037] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:46.310385 2734 raft_consensus.cc:504] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.310703 2734 leader_election.cc:288] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 election: Requested vote from peers 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:46.311015 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e19f3db2d74413b8d3620072a8c61f7" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae06d1b9c07485185beda9112176f6b"
I20260620 08:54:46.311123 2277 raft_consensus.cc:3037] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:46.311403 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e19f3db2d74413b8d3620072a8c61f7" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a"
I20260620 08:54:46.311519 2563 raft_consensus.cc:3037] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:46.312209 2277 raft_consensus.cc:2454] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 1.
I20260620 08:54:46.312516 2357 leader_election.cc:302] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae06d1b9c07485185beda9112176f6b, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:46.312610 2563 raft_consensus.cc:2454] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 1.
I20260620 08:54:46.312742 2734 raft_consensus.cc:2781] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:46.312949 2734 raft_consensus.cc:686] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 1 LEADER]: Becoming Leader. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Running, Role: LEADER
I20260620 08:54:46.313205 2734 consensus_queue.cc:237] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.315373 2125 catalog_manager.cc:5495] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead reported cstate change: term changed from 0 to 1, leader changed from <none> to c3f037dc24004201bd934e2ec88d1ead (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c3f037dc24004201bd934e2ec88d1ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } health_report { overall_health: UNKNOWN } } }
I20260620 08:54:46.341198 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.337407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.0.98.2:36393)
0620 08:54:46.337666 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:46.337681 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:46.337692 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:54:46.337757 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:46.337759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:46.337817 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:46.337921 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:46.338200 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.338749 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.339574 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.339903 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.339985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.340279 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:46.340822 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:46.340936 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:46.341015 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:54:46.341068 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260620 08:54:46.351277 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.347967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50861 (local address 127.0.98.1:42881)
0620 08:54:46.348086 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:46.348090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:46.348148 (+ 58us) server_negotiation.cc:407] Connection header received
0620 08:54:46.348252 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:46.348254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:46.348297 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:46.348355 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:46.348672 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.349215 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.350214 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.350429 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.350514 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.350858 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:46.350928 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:46.350987 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:46.351125 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 08:54:46.351171 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:54:46.351311 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.347750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.98.2:50861)
0620 08:54:46.348087 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:46.348100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:46.348187 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:46.348366 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:46.348368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:46.348373 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:46.348541 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.348556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.349321 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.349323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.350081 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.350091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.350563 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.350570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.350658 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.350699 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:46.351033 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:46.351040 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:54:46.351113 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:54:46.351185 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260620 08:54:46.356837 2563 raft_consensus.cc:1261] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Refusing update from remote peer c3f037dc24004201bd934e2ec88d1ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:46.357128 2277 raft_consensus.cc:1261] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Refusing update from remote peer c3f037dc24004201bd934e2ec88d1ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:46.357344 2739 consensus_queue.cc:1035] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:46.357491 2734 consensus_queue.cc:1035] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:46.374089 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.370982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59533 (local address 127.0.98.2:36393)
0620 08:54:46.371076 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:46.371080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:46.371093 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:54:46.371204 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:46.371207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:46.371250 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:46.371313 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:46.371641 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.372253 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.373052 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.373317 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.373409 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.373682 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:46.373749 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:46.373823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:46.373912 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 08:54:46.373969 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:54:46.374115 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:46.370766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36393 (local address 127.0.98.1:59533)
0620 08:54:46.371009 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:46.371023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:46.371108 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:46.371411 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:46.371414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:46.371423 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:54:46.371517 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.371523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.372365 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.372367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.372962 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:46.372968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:46.373444 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:46.373447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:46.373535 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:46.373567 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:54:46.373829 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:54:46.373836 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:54:46.373902 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:54:46.373983 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260620 08:54:46.396824 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:46.396999 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:46.399212 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Sending DeleteTablet for 3 replicas of tablet 6e19f3db2d74413b8d3620072a8c61f7
I20260620 08:54:46.399773 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e19f3db2d74413b8d3620072a8c61f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59698
I20260620 08:54:46.399773 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e19f3db2d74413b8d3620072a8c61f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34236
I20260620 08:54:46.399981 2717 tablet_replica.cc:330] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: stopping tablet replica
I20260620 08:54:46.399957 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e19f3db2d74413b8d3620072a8c61f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59520
I20260620 08:54:46.400089 2717 raft_consensus.cc:2229] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:46.400282 2720 tablet_replica.cc:330] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: stopping tablet replica
I20260620 08:54:46.400359 2717 raft_consensus.cc:2258] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:46.400415 2720 raft_consensus.cc:2229] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:46.400517 2720 raft_consensus.cc:2258] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:46.400667 2718 tablet_replica.cc:330] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: stopping tablet replica
I20260620 08:54:46.400771 2718 raft_consensus.cc:2229] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:46.400871 2718 raft_consensus.cc:2258] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:46.400923 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:54:46.401208 2717 ts_tablet_manager.cc:1905] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:54:46.401443 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:46.401754 2718 ts_tablet_manager.cc:1905] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:46.403352 2717 ts_tablet_manager.cc:1918] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:46.403400 2717 log.cc:1196] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/6e19f3db2d74413b8d3620072a8c61f7
I20260620 08:54:46.403554 2717 ts_tablet_manager.cc:1939] T 6e19f3db2d74413b8d3620072a8c61f7 P c3f037dc24004201bd934e2ec88d1ead: Deleting consensus metadata
I20260620 08:54:46.404525 2114 catalog_manager.cc:4841] TS c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393): tablet 6e19f3db2d74413b8d3620072a8c61f7 (table test-table [id=72f2e123fa2243a1a09af1443a97ca60]) successfully deleted
I20260620 08:54:46.404618 2718 ts_tablet_manager.cc:1918] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:46.404688 2718 log.cc:1196] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/6e19f3db2d74413b8d3620072a8c61f7
I20260620 08:54:46.405534 2718 ts_tablet_manager.cc:1939] T 6e19f3db2d74413b8d3620072a8c61f7 P 4ae06d1b9c07485185beda9112176f6b: Deleting consensus metadata
I20260620 08:54:46.406330 2113 catalog_manager.cc:4841] TS 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881): tablet 6e19f3db2d74413b8d3620072a8c61f7 (table test-table [id=72f2e123fa2243a1a09af1443a97ca60]) successfully deleted
I20260620 08:54:46.406375 2720 ts_tablet_manager.cc:1905] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:46.407814 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 548bc8e67829494a9ed3c826a6b0fc78 (DEFAULT_TABLE table=test-table [id=d3831fefa58b4c7797c2a9fcaebd64f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:46.407804 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 548bc8e67829494a9ed3c826a6b0fc78 (DEFAULT_TABLE table=test-table [id=d3831fefa58b4c7797c2a9fcaebd64f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:46.408046 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548bc8e67829494a9ed3c826a6b0fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:46.408058 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548bc8e67829494a9ed3c826a6b0fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:46.408742 2720 ts_tablet_manager.cc:1918] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:54:46.408813 2720 log.cc:1196] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/6e19f3db2d74413b8d3620072a8c61f7
I20260620 08:54:46.409029 2720 ts_tablet_manager.cc:1939] T 6e19f3db2d74413b8d3620072a8c61f7 P 382825d452d4472ca6876c0a8565d69a: Deleting consensus metadata
I20260620 08:54:46.409577 2112 catalog_manager.cc:4841] TS 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565): tablet 6e19f3db2d74413b8d3620072a8c61f7 (table test-table [id=72f2e123fa2243a1a09af1443a97ca60]) successfully deleted
I20260620 08:54:46.410079 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 548bc8e67829494a9ed3c826a6b0fc78 (DEFAULT_TABLE table=test-table [id=d3831fefa58b4c7797c2a9fcaebd64f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:46.410223 2730 tablet_bootstrap.cc:492] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Bootstrap starting.
I20260620 08:54:46.410298 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548bc8e67829494a9ed3c826a6b0fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:46.411024 2730 tablet_bootstrap.cc:654] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:46.412065 2729 tablet_bootstrap.cc:492] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b: Bootstrap starting.
I20260620 08:54:46.412156 2730 tablet_bootstrap.cc:492] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: No bootstrap required, opened a new log
I20260620 08:54:46.412194 2728 tablet_bootstrap.cc:492] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Bootstrap starting.
I20260620 08:54:46.412245 2730 ts_tablet_manager.cc:1397] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:46.412624 2730 raft_consensus.cc:348] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.412814 2729 tablet_bootstrap.cc:654] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:46.412807 2730 raft_consensus.cc:374] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:46.412874 2730 raft_consensus.cc:729] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382825d452d4472ca6876c0a8565d69a, State: Initialized, Role: FOLLOWER
I20260620 08:54:46.413004 2728 tablet_bootstrap.cc:654] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:46.413112 2730 consensus_queue.cc:260] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.413471 2730 ts_tablet_manager.cc:1428] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:46.414122 2728 tablet_bootstrap.cc:492] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: No bootstrap required, opened a new log
I20260620 08:54:46.414209 2728 ts_tablet_manager.cc:1397] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:46.414620 2728 raft_consensus.cc:348] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.414729 2728 raft_consensus.cc:374] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:46.414762 2728 raft_consensus.cc:729] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Initialized, Role: FOLLOWER
I20260620 08:54:46.414881 2728 consensus_queue.cc:260] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.415128 2728 ts_tablet_manager.cc:1428] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:54:46.416066 2729 tablet_bootstrap.cc:492] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b: No bootstrap required, opened a new log
I20260620 08:54:46.416195 2729 ts_tablet_manager.cc:1397] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 08:54:46.416592 2729 raft_consensus.cc:348] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.416697 2729 raft_consensus.cc:374] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:46.416729 2729 raft_consensus.cc:729] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae06d1b9c07485185beda9112176f6b, State: Initialized, Role: FOLLOWER
I20260620 08:54:46.416849 2729 consensus_queue.cc:260] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.417106 2729 ts_tablet_manager.cc:1428] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:54:46.550614 2739 raft_consensus.cc:482] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:46.550763 2739 raft_consensus.cc:504] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.551170 2739 leader_election.cc:288] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:46.551656 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548bc8e67829494a9ed3c826a6b0fc78" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae06d1b9c07485185beda9112176f6b" is_pre_election: true
I20260620 08:54:46.551656 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548bc8e67829494a9ed3c826a6b0fc78" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a" is_pre_election: true
I20260620 08:54:46.551805 2563 raft_consensus.cc:2454] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 0.
I20260620 08:54:46.551817 2277 raft_consensus.cc:2454] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 0.
I20260620 08:54:46.552078 2356 leader_election.cc:302] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382825d452d4472ca6876c0a8565d69a, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:46.552233 2739 raft_consensus.cc:2781] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:46.552300 2739 raft_consensus.cc:482] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:46.552346 2739 raft_consensus.cc:3037] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:46.553260 2739 raft_consensus.cc:504] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.553573 2739 leader_election.cc:288] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 election: Requested vote from peers 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881), 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565)
I20260620 08:54:46.553906 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548bc8e67829494a9ed3c826a6b0fc78" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382825d452d4472ca6876c0a8565d69a"
I20260620 08:54:46.553932 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548bc8e67829494a9ed3c826a6b0fc78" candidate_uuid: "c3f037dc24004201bd934e2ec88d1ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae06d1b9c07485185beda9112176f6b"
I20260620 08:54:46.554018 2277 raft_consensus.cc:3037] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:46.554024 2563 raft_consensus.cc:3037] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:46.554853 2277 raft_consensus.cc:2454] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 1.
I20260620 08:54:46.554936 2563 raft_consensus.cc:2454] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f037dc24004201bd934e2ec88d1ead in term 1.
I20260620 08:54:46.555157 2357 leader_election.cc:302] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae06d1b9c07485185beda9112176f6b, c3f037dc24004201bd934e2ec88d1ead; no voters:
I20260620 08:54:46.555317 2739 raft_consensus.cc:2781] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:46.555414 2739 raft_consensus.cc:686] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 1 LEADER]: Becoming Leader. State: Replica: c3f037dc24004201bd934e2ec88d1ead, State: Running, Role: LEADER
I20260620 08:54:46.555553 2739 consensus_queue.cc:237] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260620 08:54:46.557123 2125 catalog_manager.cc:5495] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead reported cstate change: term changed from 0 to 1, leader changed from <none> to c3f037dc24004201bd934e2ec88d1ead (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c3f037dc24004201bd934e2ec88d1ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3f037dc24004201bd934e2ec88d1ead" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } health_report { overall_health: UNKNOWN } } }
I20260620 08:54:46.578826 2563 raft_consensus.cc:1261] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Refusing update from remote peer c3f037dc24004201bd934e2ec88d1ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:46.578841 2277 raft_consensus.cc:1261] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Refusing update from remote peer c3f037dc24004201bd934e2ec88d1ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:46.579167 2734 consensus_queue.cc:1035] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "382825d452d4472ca6876c0a8565d69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:46.579376 2739 consensus_queue.cc:1035] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae06d1b9c07485185beda9112176f6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:54:46.593007 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 08:54:48.145502 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:48.145645 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:48.147697 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e94064ca2cc424c90ef6f2657ce5469: Sending DeleteTablet for 3 replicas of tablet 548bc8e67829494a9ed3c826a6b0fc78
I20260620 08:54:48.148211 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 548bc8e67829494a9ed3c826a6b0fc78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59698
I20260620 08:54:48.148226 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 548bc8e67829494a9ed3c826a6b0fc78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34236
I20260620 08:54:48.148305 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 548bc8e67829494a9ed3c826a6b0fc78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59520
I20260620 08:54:48.148516 2781 tablet_replica.cc:330] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b: stopping tablet replica
I20260620 08:54:48.148581 2782 tablet_replica.cc:330] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: stopping tablet replica
I20260620 08:54:48.148638 2781 raft_consensus.cc:2229] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:48.148699 2782 raft_consensus.cc:2229] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:48.148744 2781 raft_consensus.cc:2258] T 548bc8e67829494a9ed3c826a6b0fc78 P 4ae06d1b9c07485185beda9112176f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:48.148804 2782 raft_consensus.cc:2258] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:48.148931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 2199
I20260620 08:54:48.149519 2780 tablet_replica.cc:330] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: stopping tablet replica
I20260620 08:54:48.149655 2780 raft_consensus.cc:2229] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:48.149837 2780 raft_consensus.cc:2258] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:48.150367 2780 ts_tablet_manager.cc:1905] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:48.152875 2780 ts_tablet_manager.cc:1918] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:54:48.152938 2780 log.cc:1196] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/548bc8e67829494a9ed3c826a6b0fc78
I20260620 08:54:48.153175 2780 ts_tablet_manager.cc:1939] T 548bc8e67829494a9ed3c826a6b0fc78 P c3f037dc24004201bd934e2ec88d1ead: Deleting consensus metadata
I20260620 08:54:48.153769 2114 catalog_manager.cc:4841] TS c3f037dc24004201bd934e2ec88d1ead (127.0.98.2:36393): tablet 548bc8e67829494a9ed3c826a6b0fc78 (table test-table [id=d3831fefa58b4c7797c2a9fcaebd64f4]) successfully deleted
I20260620 08:54:48.156036 2782 ts_tablet_manager.cc:1905] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:54:48.156509 2113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42881 (error 108)
I20260620 08:54:48.156584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 2339
I20260620 08:54:48.158509 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:48.157860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.0.1:34268)
0620 08:54:48.158352 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:48.158428 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":98,"threads_started":1}
W20260620 08:54:48.158598 2113 catalog_manager.cc:4584] TS 4ae06d1b9c07485185beda9112176f6b (127.0.98.1:42881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 548bc8e67829494a9ed3c826a6b0fc78: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42881: connect: Connection refused (error 111)
I20260620 08:54:48.159262 2782 ts_tablet_manager.cc:1918] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:54:48.159341 2782 log.cc:1196] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/548bc8e67829494a9ed3c826a6b0fc78
I20260620 08:54:48.159575 2782 ts_tablet_manager.cc:1939] T 548bc8e67829494a9ed3c826a6b0fc78 P 382825d452d4472ca6876c0a8565d69a: Deleting consensus metadata
I20260620 08:54:48.160660 2112 catalog_manager.cc:4841] TS 382825d452d4472ca6876c0a8565d69a (127.0.98.3:36565): tablet 548bc8e67829494a9ed3c826a6b0fc78 (table test-table [id=d3831fefa58b4c7797c2a9fcaebd64f4]) successfully deleted
I20260620 08:54:48.164039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 2479
I20260620 08:54:48.169603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 2095
2026-06-20T08:54:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6312 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:48 dist-test-slave-fb1v krb5kdc[2787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:48 dist-test-slave-fb1v krb5kdc[2787](info): set up 2 sockets
Jun 20 08:54:48 dist-test-slave-fb1v krb5kdc[2787](info): commencing operation
krb5kdc: starting...
W20260620 08:54:49.127341 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:49.243187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:49.358403 2800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:49.358703 2800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:49.358747 2800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:49.362814 2800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:49.362871 2800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:49.362890 2800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:49.362905 2800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:49.362919 2800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:49.368422 2800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:49.369431 2800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:49.370371 2800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:49.378697 2805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:49.378726 2808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:49.378731 2806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:49.378847 2800 server_base.cc:1034] running on GCE node
I20260620 08:54:49.379544 2800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:49.380602 2800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:49.381772 2800 hybrid_clock.cc:648] HybridClock initialized: now 1781945689381717 us; error 68 us; skew 500 ppm
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:49.384409 2800 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:49.385224 2800 webserver.cc:460] Webserver started at http://127.0.98.62:40473/ using document root <none> and password file <none>
I20260620 08:54:49.385736 2800 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:49.385800 2800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:49.385972 2800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:49.387944 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "2b733ec746d7430194b8ca25e23ffd3b"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "366c787776c6de60b14c0caf67e385a4"
server_key_iv: "a315939c847061f104365bff0e321a34"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.388450 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b733ec746d7430194b8ca25e23ffd3b"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "366c787776c6de60b14c0caf67e385a4"
server_key_iv: "a315939c847061f104365bff0e321a34"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.391970 2800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:49.394560 2815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.395857 2800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:49.395982 2800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "2b733ec746d7430194b8ca25e23ffd3b"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "366c787776c6de60b14c0caf67e385a4"
server_key_iv: "a315939c847061f104365bff0e321a34"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.396075 2800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:49.413323 2800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:49.415966 2800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:49.416128 2800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:49.422410 2800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41127
I20260620 08:54:49.422456 2867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41127 every 8 connection(s)
I20260620 08:54:49.423300 2800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:49.425835 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:49.429796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 2800
I20260620 08:54:49.429889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:49.430115 392 external_mini_cluster.cc:1428] Setting key 1c46525d5cecf44a9b6626854dc9af8e
I20260620 08:54:49.431257 2868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b: Bootstrap starting.
I20260620 08:54:49.433525 2868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:49.434285 2868 log.cc:824] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:49.436233 2868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b: No bootstrap required, opened a new log
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:49.440227 2868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b733ec746d7430194b8ca25e23ffd3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260620 08:54:49.440441 2868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:49.440491 2868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b733ec746d7430194b8ca25e23ffd3b, State: Initialized, Role: FOLLOWER
I20260620 08:54:49.440909 2868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b733ec746d7430194b8ca25e23ffd3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260620 08:54:49.441039 2868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:49.441100 2868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:49.441192 2868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:49.441984 2868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b733ec746d7430194b8ca25e23ffd3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260620 08:54:49.442283 2868 leader_election.cc:302] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b733ec746d7430194b8ca25e23ffd3b; no voters:
I20260620 08:54:49.442576 2868 leader_election.cc:288] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:49.442654 2873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:49.442876 2873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [term 1 LEADER]: Becoming Leader. State: Replica: 2b733ec746d7430194b8ca25e23ffd3b, State: Running, Role: LEADER
I20260620 08:54:49.443222 2873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b733ec746d7430194b8ca25e23ffd3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260620 08:54:49.443853 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:49.431433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.0.98.62:41127)
0620 08:54:49.431806 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:49.431821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:49.431855 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:54:49.432459 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:49.432474 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:49.432781 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:49.433039 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:49.433907 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.434911 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.436047 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.436541 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.436654 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:49.439375 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:49.439380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:49.439395 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:49.439420 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:49.441056 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:49.441463 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:49.441469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:49.441474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:49.441578 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:49.441815 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:49.441818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:49.441820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:49.442215 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:49.442392 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:49.442657 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 08:54:49.442949 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":120,"threads_started":1}
I20260620 08:54:49.443905 2868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:49.444808 2875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b733ec746d7430194b8ca25e23ffd3b. Latest consensus state: current_term: 1 leader_uuid: "2b733ec746d7430194b8ca25e23ffd3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b733ec746d7430194b8ca25e23ffd3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } }
I20260620 08:54:49.444882 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b733ec746d7430194b8ca25e23ffd3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b733ec746d7430194b8ca25e23ffd3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } }
I20260620 08:54:49.444943 2875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:49.444943 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:49.446115 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:49.449155 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:49.454021 2879 catalog_manager.cc:1347] Generated new cluster ID: d884beca7e7b4ccfba1d144eec3ec5db
I20260620 08:54:49.454079 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:49.460994 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:49.461897 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:49.467938 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b733ec746d7430194b8ca25e23ffd3b: Generated new TSK 0
I20260620 08:54:49.468648 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:49.499310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:49.605487 2894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:49.605724 2894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:49.605764 2894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:49.610474 2894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:49.610531 2894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:49.610589 2894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:49.615746 2894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:49.616663 2894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:49.617568 2894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:49.626925 2902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:49.627013 2900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:49.627158 2894 server_base.cc:1034] running on GCE node
W20260620 08:54:49.626942 2899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:49.627662 2894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:49.628211 2894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:49.629381 2894 hybrid_clock.cc:648] HybridClock initialized: now 1781945689629361 us; error 32 us; skew 500 ppm
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:49.632057 2894 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:49.632911 2894 webserver.cc:460] Webserver started at http://127.0.98.1:40367/ using document root <none> and password file <none>
I20260620 08:54:49.633432 2894 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:49.633502 2894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:49.633673 2894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:49.635650 2894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "a3ff9003265a4ccb8c9026e2465fbf3e"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "3b4338726c1fdf296c1f80101a4d9159"
server_key_iv: "1fa8fece3d48a4aafd27f354735da8ed"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.636169 2894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a3ff9003265a4ccb8c9026e2465fbf3e"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "3b4338726c1fdf296c1f80101a4d9159"
server_key_iv: "1fa8fece3d48a4aafd27f354735da8ed"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.639684 2894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:49.641924 2909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.643044 2894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:49.643146 2894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "a3ff9003265a4ccb8c9026e2465fbf3e"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "3b4338726c1fdf296c1f80101a4d9159"
server_key_iv: "1fa8fece3d48a4aafd27f354735da8ed"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.643234 2894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:49.666589 2894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:49.669852 2894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:49.670035 2894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:49.670605 2894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:49.671376 2894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:49.671422 2894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.671473 2894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:49.671502 2894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.680092 2894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37183
I20260620 08:54:49.680115 3022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37183 every 8 connection(s)
I20260620 08:54:49.681046 2894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:49.684932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 2894
I20260620 08:54:49.685040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:49.685271 392 external_mini_cluster.cc:1428] Setting key 116912584635f5034635aa3a3067bb73
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:54:49.694753 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:49.682733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56425 (local address 127.0.98.62:41127)
0620 08:54:49.682862 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:49.682866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:49.683778 (+ 912us) server_negotiation.cc:407] Connection header received
0620 08:54:49.685148 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:49.685151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:49.685204 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:49.685273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:49.686486 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.687001 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.688348 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.688582 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.688676 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:49.691244 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:49.691248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:49.691251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:49.691280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:49.692634 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:49.693205 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:49.693208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:49.693209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:49.693292 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:49.693663 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:49.693666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:49.693668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:49.693872 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:49.693974 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:49.694461 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 08:54:49.694589 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:54:49.695676 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:49.683144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:56425)
0620 08:54:49.683565 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:49.683685 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:49.684915 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:49.685453 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:49.685460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:49.685834 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:49.686357 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:49.686375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.687111 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.687114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:49.688217 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:49.688228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.688730 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.688734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:49.688926 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:49.689519 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:49.689538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:49.691082 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:49.692746 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:49.692752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:49.692777 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:49.693095 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:49.693393 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:49.693396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:49.693398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:49.693570 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:49.693968 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:49.693974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:49.694233 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 08:54:49.694407 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 08:54:49.694754 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:54:49.696822 3023 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260620 08:54:49.697063 3023 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:49.697580 3023 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260620 08:54:49.699060 2832 ts_manager.cc:194] Registered new tserver with Master: a3ff9003265a4ccb8c9026e2465fbf3e (127.0.98.1:37183)
I20260620 08:54:49.700201 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:49.710659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:49.815114 3028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:49.815362 3028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:49.815428 3028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:49.819384 3028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:49.819445 3028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:49.819523 3028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:49.824586 3028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:49.825533 3028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:49.826481 3028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:49.835441 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:49.835500 3033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:49.835510 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:49.835582 3028 server_base.cc:1034] running on GCE node
I20260620 08:54:49.836094 3028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:49.836664 3028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:49.837831 3028 hybrid_clock.cc:648] HybridClock initialized: now 1781945689837814 us; error 39 us; skew 500 ppm
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:49.840569 3028 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:49.841382 3028 webserver.cc:460] Webserver started at http://127.0.98.2:46755/ using document root <none> and password file <none>
I20260620 08:54:49.841885 3028 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:49.841946 3028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:49.842118 3028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:49.844192 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "3cda332e43524856b7051564618cc00b"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "c91dfd7a997f932e59ccd8e1a3f40ae9"
server_key_iv: "8a04c8ef1c3fc242d1997dd5b0ed4fa6"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.844712 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3cda332e43524856b7051564618cc00b"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "c91dfd7a997f932e59ccd8e1a3f40ae9"
server_key_iv: "8a04c8ef1c3fc242d1997dd5b0ed4fa6"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.848774 3028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 08:54:49.851074 3043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.852357 3028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:49.852451 3028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "3cda332e43524856b7051564618cc00b"
format_stamp: "Formatted at 2026-06-20 08:54:49 on dist-test-slave-fb1v"
server_key: "c91dfd7a997f932e59ccd8e1a3f40ae9"
server_key_iv: "8a04c8ef1c3fc242d1997dd5b0ed4fa6"
server_key_version: "encryptionkey@0"
I20260620 08:54:49.852551 3028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:49.866513 3028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:49.869387 3028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:49.869550 3028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:49.870098 3028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:49.870842 3028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:49.870888 3028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.870925 3028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:49.870952 3028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:49.878930 3028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38177
I20260620 08:54:49.878954 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38177 every 8 connection(s)
I20260620 08:54:49.879828 3028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:49.886344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3028
I20260620 08:54:49.886456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:49.886759 392 external_mini_cluster.cc:1428] Setting key e337d750b355b90473e6f2cb89de20c3
Jun 20 08:54:49 dist-test-slave-fb1v krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:49.893572 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:49.881396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47365 (local address 127.0.98.62:41127)
0620 08:54:49.881511 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:49.881516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:49.882323 (+ 807us) server_negotiation.cc:407] Connection header received
0620 08:54:49.883666 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:49.883669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:49.883761 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:49.883834 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:49.884950 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.885455 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.886739 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.887004 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.887103 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:49.889791 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:49.889796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:49.889799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:49.889828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:49.891486 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:49.892021 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:49.892025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:49.892027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:49.892124 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:49.892483 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:49.892486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:49.892488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:49.892732 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:49.892850 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:49.893267 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 08:54:49.893387 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:54:49.894255 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:49.881709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:47365)
0620 08:54:49.882161 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:49.882201 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:49.883448 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:49.884004 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:49.884011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:49.884348 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:49.884811 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:49.884824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.885568 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.885571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:49.886627 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:49.886637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:49.887144 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:49.887147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:49.887318 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:49.887917 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:49.887938 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:49.889567 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:49.891597 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:49.891619 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:49.891635 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:49.891906 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:49.892254 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:49.892256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:49.892257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:49.892386 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:49.892836 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:49.892840 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:49.893046 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 08:54:49.893191 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 08:54:49.893428 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260620 08:54:49.895277 3157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260620 08:54:49.895526 3157 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:49.896139 3157 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:54:49.897222 2832 ts_manager.cc:194] Registered new tserver with Master: 3cda332e43524856b7051564618cc00b (127.0.98.2:38177)
I20260620 08:54:49.897779 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:49.912320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:50.023393 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:50.023694 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:50.023746 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:50.027840 3162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:50.027901 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:50.027959 3162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:50.033147 3162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:50.034132 3162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:50.035069 3162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:50.044416 3167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:50.044476 3168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:50.044588 3162 server_base.cc:1034] running on GCE node
W20260620 08:54:50.044785 3170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:50.045151 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:50.045771 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:50.046937 3162 hybrid_clock.cc:648] HybridClock initialized: now 1781945690046926 us; error 24 us; skew 500 ppm
Jun 20 08:54:50 dist-test-slave-fb1v krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:50.049594 3162 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:50.050433 3162 webserver.cc:460] Webserver started at http://127.0.98.3:34673/ using document root <none> and password file <none>
I20260620 08:54:50.050966 3162 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:50.051026 3162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:50.051201 3162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:50.053205 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "2be7a3d0d43e47ab9cd5b5e91e6da8fb"
format_stamp: "Formatted at 2026-06-20 08:54:50 on dist-test-slave-fb1v"
server_key: "e0b6fff931b718d93f1313eda5989b09"
server_key_iv: "6c70b45b7479ada9ac40afaaf8d60810"
server_key_version: "encryptionkey@0"
I20260620 08:54:50.053714 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2be7a3d0d43e47ab9cd5b5e91e6da8fb"
format_stamp: "Formatted at 2026-06-20 08:54:50 on dist-test-slave-fb1v"
server_key: "e0b6fff931b718d93f1313eda5989b09"
server_key_iv: "6c70b45b7479ada9ac40afaaf8d60810"
server_key_version: "encryptionkey@0"
I20260620 08:54:50.057288 3162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:50.059677 3177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:50.060868 3162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:50.060973 3162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "2be7a3d0d43e47ab9cd5b5e91e6da8fb"
format_stamp: "Formatted at 2026-06-20 08:54:50 on dist-test-slave-fb1v"
server_key: "e0b6fff931b718d93f1313eda5989b09"
server_key_iv: "6c70b45b7479ada9ac40afaaf8d60810"
server_key_version: "encryptionkey@0"
I20260620 08:54:50.061070 3162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:50.073191 3162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:50.076069 3162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:50.076232 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:50.076771 3162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:50.077526 3162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:50.077571 3162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:50.077620 3162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:50.077646 3162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:50.085976 3162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44067
I20260620 08:54:50.086007 3290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44067 every 8 connection(s)
I20260620 08:54:50.086784 3162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:50.088127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3162
I20260620 08:54:50.088207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:50.088483 392 external_mini_cluster.cc:1428] Setting key ca9cd5d31b9d32f3153939c78fb2b123
Jun 20 08:54:50 dist-test-slave-fb1v krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:50.099992 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:50.088541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46327 (local address 127.0.98.62:41127)
0620 08:54:50.088676 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:50.088681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:50.089545 (+ 864us) server_negotiation.cc:407] Connection header received
0620 08:54:50.091077 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:50.091080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:50.091123 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:50.091191 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:50.092418 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:50.092959 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:50.094198 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:50.094439 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:50.094526 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:50.096818 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:50.096820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:50.096822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:50.096845 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:50.098146 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:50.098584 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:50.098586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:50.098588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:50.098681 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:50.099010 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:50.099014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:50.099015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:50.099194 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:50.099280 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:50.099768 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 08:54:50.099861 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 08:54:50.100824 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:50.088934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:46327)
0620 08:54:50.089426 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:50.089466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:50.090880 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:50.091379 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:50.091386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:50.091797 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:50.092285 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:50.092298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:50.093090 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:50.093092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:50.094030 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:50.094040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:50.094591 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:50.094594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:50.094742 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:50.095287 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:50.095303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:50.096665 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:50.098247 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:50.098251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:50.098262 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:50.098502 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:50.098755 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:50.098757 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:50.098758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:50.098905 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:50.099301 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:50.099306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:50.099527 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 08:54:50.099725 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 08:54:50.100047 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260620 08:54:50.102118 3291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260620 08:54:50.102349 3291 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:50.102813 3291 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260620 08:54:50.103801 2832 ts_manager.cc:194] Registered new tserver with Master: 2be7a3d0d43e47ab9cd5b5e91e6da8fb (127.0.98.3:44067)
I20260620 08:54:50.104322 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46327
I20260620 08:54:50.113796 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:54:50.125473 2871 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:54:50.124156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.0.98.62:41127)
0620 08:54:50.124303 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:50.124307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:50.124398 (+ 91us) server_negotiation.cc:407] Connection header received
0620 08:54:50.124532 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:50.124536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:50.124588 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:50.124658 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:50.125367 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40564: BlockingRecv error: recv got EOF from 127.0.0.1:40564 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:54:50.126658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 2894
I20260620 08:54:50.131254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3028
I20260620 08:54:50.135903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3162
I20260620 08:54:50.140196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 2800
2026-06-20T08:54:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1969 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:50 dist-test-slave-fb1v krb5kdc[3303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:50 dist-test-slave-fb1v krb5kdc[3303](info): set up 2 sockets
Jun 20 08:54:50 dist-test-slave-fb1v krb5kdc[3303](info): commencing operation
krb5kdc: starting...
W20260620 08:54:51.095307 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:51 dist-test-slave-fb1v krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:51.209960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44339
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:51.317414 3316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:51.317677 3316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:51.317728 3316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:51.321895 3316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:51.321965 3316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:51.321995 3316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:51.322016 3316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:51.322037 3316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:51.327258 3316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:51.328272 3316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:51.329221 3316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:51.337247 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:51.337257 3321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:51.337441 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:51.337461 3316 server_base.cc:1034] running on GCE node
I20260620 08:54:51.338174 3316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:51.339182 3316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:51.340348 3316 hybrid_clock.cc:648] HybridClock initialized: now 1781945691340330 us; error 34 us; skew 500 ppm
Jun 20 08:54:51 dist-test-slave-fb1v krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:51.342990 3316 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:51.343855 3316 webserver.cc:460] Webserver started at http://127.0.98.62:35561/ using document root <none> and password file <none>
I20260620 08:54:51.344358 3316 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:51.344422 3316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:51.344612 3316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:51.346599 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "8f100121fe524c82b0ed076a39c7a4b2"
format_stamp: "Formatted at 2026-06-20 08:54:51 on dist-test-slave-fb1v"
server_key: "032cadfe48a1c229514499bf461ee05f"
server_key_iv: "4410eac21e15b50a6725ad6826b48ba1"
server_key_version: "encryptionkey@0"
I20260620 08:54:51.347407 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f100121fe524c82b0ed076a39c7a4b2"
format_stamp: "Formatted at 2026-06-20 08:54:51 on dist-test-slave-fb1v"
server_key: "032cadfe48a1c229514499bf461ee05f"
server_key_iv: "4410eac21e15b50a6725ad6826b48ba1"
server_key_version: "encryptionkey@0"
I20260620 08:54:51.352298 3316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:54:51.355155 3331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:51.356666 3316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:51.356786 3316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "8f100121fe524c82b0ed076a39c7a4b2"
format_stamp: "Formatted at 2026-06-20 08:54:51 on dist-test-slave-fb1v"
server_key: "032cadfe48a1c229514499bf461ee05f"
server_key_iv: "4410eac21e15b50a6725ad6826b48ba1"
server_key_version: "encryptionkey@0"
I20260620 08:54:51.356884 3316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:51.373780 3316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:51.376832 3316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:51.376983 3316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:51.383098 3316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44339
I20260620 08:54:51.383351 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44339 every 8 connection(s)
I20260620 08:54:51.384019 3316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:51.385576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3316
I20260620 08:54:51.385663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:51.385910 392 external_mini_cluster.cc:1428] Setting key 290687d4628be8037b6eb3956c34ca75
I20260620 08:54:51.386896 3384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:51.393194 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2: Bootstrap starting.
Jun 20 08:54:51 dist-test-slave-fb1v krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:51.395463 3384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:51.396313 3384 log.cc:824] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:51.398435 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2: No bootstrap required, opened a new log
I20260620 08:54:51.400454 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:51.387148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.98.62:44339)
0620 08:54:51.387549 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:51.387567 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:51.387639 (+ 72us) server_negotiation.cc:407] Connection header received
0620 08:54:51.388476 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:51.388506 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:51.388955 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:51.389342 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:51.389986 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:51.391425 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:51.392339 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:51.392975 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:51.393130 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:51.395491 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:51.395497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:51.395512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:51.395543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:51.397405 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:51.397796 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:51.397801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:51.397806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:51.397934 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:51.398146 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:51.398149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:51.398151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:51.398641 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:51.398821 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:51.399014 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 08:54:51.399298 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260620 08:54:51.402032 3384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f100121fe524c82b0ed076a39c7a4b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44339 } }
I20260620 08:54:51.402320 3384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:51.402392 3384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f100121fe524c82b0ed076a39c7a4b2, State: Initialized, Role: FOLLOWER
I20260620 08:54:51.402863 3384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f100121fe524c82b0ed076a39c7a4b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44339 } }
I20260620 08:54:51.403033 3384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:51.403110 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:51.403141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:54:51.403206 3384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:51.404099 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f100121fe524c82b0ed076a39c7a4b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44339 } }
I20260620 08:54:51.404402 3384 leader_election.cc:302] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f100121fe524c82b0ed076a39c7a4b2; no voters:
I20260620 08:54:51.404712 3384 leader_election.cc:288] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:51.404816 3390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:51.405056 3390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [term 1 LEADER]: Becoming Leader. State: Replica: 8f100121fe524c82b0ed076a39c7a4b2, State: Running, Role: LEADER
I20260620 08:54:51.405378 3390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f100121fe524c82b0ed076a39c7a4b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44339 } }
I20260620 08:54:51.406024 3384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:51.406862 3391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f100121fe524c82b0ed076a39c7a4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f100121fe524c82b0ed076a39c7a4b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44339 } } }
I20260620 08:54:51.407011 3391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:51.407244 3392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f100121fe524c82b0ed076a39c7a4b2. Latest consensus state: current_term: 1 leader_uuid: "8f100121fe524c82b0ed076a39c7a4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f100121fe524c82b0ed076a39c7a4b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44339 } } }
I20260620 08:54:51.407338 3399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:51.407330 3392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:51.410395 3399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:51.416265 3399 catalog_manager.cc:1347] Generated new cluster ID: 5f6b52ac051c4ae9a953fe02df9ca811
I20260620 08:54:51.416357 3399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:51.422675 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:51.417468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.98.62:44339)
0620 08:54:51.417586 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:51.417591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:51.417783 (+ 192us) server_negotiation.cc:407] Connection header received
0620 08:54:51.417889 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:51.417892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:51.417943 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:51.418034 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:51.418535 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:51.419354 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:51.420490 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:51.420843 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:51.420946 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:51.421802 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:51.421806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:51.421809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:51.422224 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:51.422257 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:51.422277 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:51.422373 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:51.422483 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:54:51.422542 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:54:51.423167 3399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:51.424106 3399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:51.435429 3399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f100121fe524c82b0ed076a39c7a4b2: Generated new TSK 0
I20260620 08:54:51.436125 3399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:54:51.445881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3316
2026-06-20T08:54:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1303 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:51 dist-test-slave-fb1v krb5kdc[3417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:51 dist-test-slave-fb1v krb5kdc[3417](info): set up 2 sockets
Jun 20 08:54:51 dist-test-slave-fb1v krb5kdc[3417](info): commencing operation
krb5kdc: starting...
W20260620 08:54:52.419332 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:52 dist-test-slave-fb1v krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:52.530341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:52.635826 3430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:52.636082 3430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:52.636121 3430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:52.640096 3430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:52.640144 3430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:52.640162 3430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:52.640177 3430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:52.640202 3430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:52.645357 3430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:52.646292 3430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:52.647230 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:52.655140 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:52.655161 3435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:52.655156 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:52.655270 3430 server_base.cc:1034] running on GCE node
I20260620 08:54:52.655963 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:52.656996 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:52.658174 3430 hybrid_clock.cc:648] HybridClock initialized: now 1781945692658159 us; error 32 us; skew 500 ppm
Jun 20 08:54:52 dist-test-slave-fb1v krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:52.660790 3430 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:52.661614 3430 webserver.cc:460] Webserver started at http://127.0.98.62:37461/ using document root <none> and password file <none>
I20260620 08:54:52.662111 3430 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:52.662164 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:52.662317 3430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:52.664292 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "d0c89020cf7f4fe594ec81613948be01"
format_stamp: "Formatted at 2026-06-20 08:54:52 on dist-test-slave-fb1v"
server_key: "c70c55d49e65329fd634e592859dde17"
server_key_iv: "97bf0a6b96cb8a05e2b9dabf810b16f3"
server_key_version: "encryptionkey@0"
I20260620 08:54:52.664774 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0c89020cf7f4fe594ec81613948be01"
format_stamp: "Formatted at 2026-06-20 08:54:52 on dist-test-slave-fb1v"
server_key: "c70c55d49e65329fd634e592859dde17"
server_key_iv: "97bf0a6b96cb8a05e2b9dabf810b16f3"
server_key_version: "encryptionkey@0"
I20260620 08:54:52.668161 3430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:52.670325 3445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:52.671505 3430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:52.671602 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "d0c89020cf7f4fe594ec81613948be01"
format_stamp: "Formatted at 2026-06-20 08:54:52 on dist-test-slave-fb1v"
server_key: "c70c55d49e65329fd634e592859dde17"
server_key_iv: "97bf0a6b96cb8a05e2b9dabf810b16f3"
server_key_version: "encryptionkey@0"
I20260620 08:54:52.671731 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:52.695364 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:52.698199 3430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:52.698348 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:52.705533 3430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34879
I20260620 08:54:52.705863 3497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34879 every 8 connection(s)
I20260620 08:54:52.706509 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:52.709213 3498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:52.714485 3498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01: Bootstrap starting.
I20260620 08:54:52.716037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3430
I20260620 08:54:52.716142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:52.716361 392 external_mini_cluster.cc:1428] Setting key ed267ffeb44f18b5fc1ecfb8afb7f43d
I20260620 08:54:52.716955 3498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:52.717782 3498 log.cc:824] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:52.719588 3498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01: No bootstrap required, opened a new log
I20260620 08:54:52.722879 3498 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c89020cf7f4fe594ec81613948be01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260620 08:54:52.723086 3498 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:52.723140 3498 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c89020cf7f4fe594ec81613948be01, State: Initialized, Role: FOLLOWER
I20260620 08:54:52.723551 3498 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c89020cf7f4fe594ec81613948be01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260620 08:54:52.723702 3498 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:52.723763 3498 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:52.723869 3498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:54:52 dist-test-slave-fb1v krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:52.724681 3498 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c89020cf7f4fe594ec81613948be01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260620 08:54:52.724998 3498 leader_election.cc:302] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c89020cf7f4fe594ec81613948be01; no voters:
I20260620 08:54:52.725328 3498 leader_election.cc:288] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:52.725438 3503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:52.725971 3503 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [term 1 LEADER]: Becoming Leader. State: Replica: d0c89020cf7f4fe594ec81613948be01, State: Running, Role: LEADER
I20260620 08:54:52.726284 3503 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c89020cf7f4fe594ec81613948be01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260620 08:54:52.726639 3498 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:52.729218 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c89020cf7f4fe594ec81613948be01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c89020cf7f4fe594ec81613948be01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } }
I20260620 08:54:52.729331 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:52.730057 3505 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c89020cf7f4fe594ec81613948be01. Latest consensus state: current_term: 1 leader_uuid: "d0c89020cf7f4fe594ec81613948be01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c89020cf7f4fe594ec81613948be01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } }
I20260620 08:54:52.730165 3505 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:52.730808 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:52.717389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.0.98.62:34879)
0620 08:54:52.717903 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:52.717930 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:52.717967 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:54:52.718706 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:52.718725 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:52.719053 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:52.719356 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:52.720129 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:52.721359 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:52.722215 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:52.722677 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:52.722817 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:52.725236 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:52.725242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:52.725261 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:52.725303 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:52.727279 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:52.727901 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:52.727907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:52.727913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:52.728062 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:52.728358 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:52.728362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:52.728364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:52.728864 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:52.729069 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:52.729477 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 08:54:52.729925 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
W20260620 08:54:52.732098 3518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:54:52.732182 3518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:54:52.732257 3519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:52.733101 3519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:52.738425 3519 catalog_manager.cc:1347] Generated new cluster ID: 93e9cac68021400db2a842ff1eefe43b
I20260620 08:54:52.738497 3519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:52.755914 3519 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:52.756834 3519 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:52.766177 3519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c89020cf7f4fe594ec81613948be01: Generated new TSK 0
I20260620 08:54:52.766800 3519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:52.796626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:52.922415 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:52.922701 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:52.922744 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:52.926774 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:52.926833 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:52.926903 3524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:52.932215 3524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:52.933264 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:52.934221 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:54:52.944392 3524 server_base.cc:1034] running on GCE node
W20260620 08:54:52.944387 3532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:52.944434 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:52.944579 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:52.944924 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:52.945479 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:52.946635 3524 hybrid_clock.cc:648] HybridClock initialized: now 1781945692946617 us; error 28 us; skew 500 ppm
Jun 20 08:54:52 dist-test-slave-fb1v krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:52.949184 3524 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:52.950027 3524 webserver.cc:460] Webserver started at http://127.0.98.1:36885/ using document root <none> and password file <none>
I20260620 08:54:52.950528 3524 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:52.950587 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:52.950758 3524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:52.952723 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "4519fb1e1efc451e9a722726837b69c8"
format_stamp: "Formatted at 2026-06-20 08:54:52 on dist-test-slave-fb1v"
server_key: "34c374d44a7bfc59ecef91a3e8c0da05"
server_key_iv: "a65095846f1ff50162e775370708a716"
server_key_version: "encryptionkey@0"
I20260620 08:54:52.953228 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4519fb1e1efc451e9a722726837b69c8"
format_stamp: "Formatted at 2026-06-20 08:54:52 on dist-test-slave-fb1v"
server_key: "34c374d44a7bfc59ecef91a3e8c0da05"
server_key_iv: "a65095846f1ff50162e775370708a716"
server_key_version: "encryptionkey@0"
I20260620 08:54:52.956710 3524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 08:54:52.959268 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:52.960464 3524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:52.960570 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "4519fb1e1efc451e9a722726837b69c8"
format_stamp: "Formatted at 2026-06-20 08:54:52 on dist-test-slave-fb1v"
server_key: "34c374d44a7bfc59ecef91a3e8c0da05"
server_key_iv: "a65095846f1ff50162e775370708a716"
server_key_version: "encryptionkey@0"
I20260620 08:54:52.960675 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:52.982656 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:52.987094 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:52.987432 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:52.988168 3524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:52.989022 3524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:52.989069 3524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:52.989123 3524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:52.989151 3524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:52.997531 3524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39801
I20260620 08:54:52.997555 3652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39801 every 8 connection(s)
I20260620 08:54:52.998360 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:53.003336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3524
I20260620 08:54:53.003484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:53.003912 392 external_mini_cluster.cc:1428] Setting key 1ee95efe6051d673c6c5bb89c2eaf02f
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:54:53.013533 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.000092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53669 (local address 127.0.98.62:34879)
0620 08:54:53.000227 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:53.000232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:53.001097 (+ 865us) server_negotiation.cc:407] Connection header received
0620 08:54:53.002441 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:53.002444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:53.002588 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:53.002687 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:53.004549 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.005251 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.006422 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.006660 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.006766 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.009523 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:53.009528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:53.009531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:53.009569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:53.011308 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.011904 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.011907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.011909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.012025 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.012339 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.012343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.012344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.012582 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:53.012690 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:53.013221 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 08:54:53.013356 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:54:53.014345 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.000529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.1:53669)
0620 08:54:53.000957 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:53.000996 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:53.002225 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:53.003137 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:53.003149 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:53.003694 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:53.004379 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:53.004397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.005384 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.005387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:53.006297 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:53.006307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.006807 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.006811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:53.006991 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.007602 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:53.007651 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:53.009343 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:53.011448 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:53.011457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:53.011471 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:53.011804 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:53.012113 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:53.012115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:53.012116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:53.012244 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:53.012707 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:53.012713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:53.012972 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 08:54:53.013126 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 08:54:53.013410 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260620 08:54:53.015659 3653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:54:53.016153 3653 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:53.016892 3653 heartbeater.cc:511] Master 127.0.98.62:34879 requested a full tablet report, sending...
I20260620 08:54:53.018710 3462 ts_manager.cc:194] Registered new tserver with Master: 4519fb1e1efc451e9a722726837b69c8 (127.0.98.1:39801)
I20260620 08:54:53.019938 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:53.032665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:53.139132 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:53.139370 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:53.139411 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:53.143381 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:53.143441 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:53.143507 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:53.148620 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:53.149622 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:53.150578 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:53.159835 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:53.159830 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:53.159907 3658 server_base.cc:1034] running on GCE node
W20260620 08:54:53.159830 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:53.160444 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:53.161096 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:53.162294 3658 hybrid_clock.cc:648] HybridClock initialized: now 1781945693162270 us; error 36 us; skew 500 ppm
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:53.165143 3658 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:53.166019 3658 webserver.cc:460] Webserver started at http://127.0.98.2:41745/ using document root <none> and password file <none>
I20260620 08:54:53.166548 3658 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:53.166610 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:53.166805 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:53.168915 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "630c2000bd2e4250b8fd0c1363ab6f47"
format_stamp: "Formatted at 2026-06-20 08:54:53 on dist-test-slave-fb1v"
server_key: "1ee30b984681258b074544a8762a74b6"
server_key_iv: "d53dd74257a859363a97d5553cbde8e9"
server_key_version: "encryptionkey@0"
I20260620 08:54:53.169435 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "630c2000bd2e4250b8fd0c1363ab6f47"
format_stamp: "Formatted at 2026-06-20 08:54:53 on dist-test-slave-fb1v"
server_key: "1ee30b984681258b074544a8762a74b6"
server_key_iv: "d53dd74257a859363a97d5553cbde8e9"
server_key_version: "encryptionkey@0"
I20260620 08:54:53.173313 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 08:54:53.175936 3673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:53.177188 3658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:54:53.177307 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "630c2000bd2e4250b8fd0c1363ab6f47"
format_stamp: "Formatted at 2026-06-20 08:54:53 on dist-test-slave-fb1v"
server_key: "1ee30b984681258b074544a8762a74b6"
server_key_iv: "d53dd74257a859363a97d5553cbde8e9"
server_key_version: "encryptionkey@0"
I20260620 08:54:53.177407 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:53.191406 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:53.195055 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:53.195250 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:53.195895 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:53.196722 3658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:53.196770 3658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:53.196822 3658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:53.196851 3658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:53.206295 3658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45301
I20260620 08:54:53.206365 3786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45301 every 8 connection(s)
I20260620 08:54:53.207242 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:53.208272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3658
I20260620 08:54:53.208365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:53.208632 392 external_mini_cluster.cc:1428] Setting key 34c921b26cab0fa12d6f6e825c005e9c
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:54:53.224771 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.209048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37031 (local address 127.0.98.62:34879)
0620 08:54:53.209168 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:53.209172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:53.210040 (+ 868us) server_negotiation.cc:407] Connection header received
0620 08:54:53.211430 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:53.211433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:53.211480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:53.211547 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:53.213056 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.213586 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.214748 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.214977 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.215066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.218081 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:53.218091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:53.218096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:53.218125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:53.222473 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.223059 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.223063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.223064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.223143 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.223457 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.223467 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.223468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.223706 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:53.223809 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:53.224516 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 08:54:53.224637 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:54:53.225955 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.209459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.2:37031)
0620 08:54:53.209902 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:53.209962 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:53.211203 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:53.211861 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:53.211871 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:53.212380 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:53.212915 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:53.212929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.213717 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.213720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:53.214630 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:53.214640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.215114 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.215117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:53.215284 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.215875 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:53.215891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:53.217766 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:53.222631 (+ 4865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:53.222639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:53.222659 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:53.222947 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:53.223226 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:53.223229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:53.223231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:53.223369 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:53.223828 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:53.223836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:53.224219 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 08:54:53.224453 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 08:54:53.224843 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
I20260620 08:54:53.227844 3787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34879
I20260620 08:54:53.228129 3787 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:53.228695 3787 heartbeater.cc:511] Master 127.0.98.62:34879 requested a full tablet report, sending...
I20260620 08:54:53.230242 3462 ts_manager.cc:194] Registered new tserver with Master: 630c2000bd2e4250b8fd0c1363ab6f47 (127.0.98.2:45301)
I20260620 08:54:53.230834 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:53.238544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:53.352352 3792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:53.352686 3792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:53.352748 3792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:53.357054 3792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:53.357113 3792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:53.357173 3792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:53.362342 3792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:53.363332 3792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:53.364368 3792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:53.373917 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:53.373914 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:53.373914 3800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:53.373975 3792 server_base.cc:1034] running on GCE node
I20260620 08:54:53.374584 3792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:53.375241 3792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:53.376403 3792 hybrid_clock.cc:648] HybridClock initialized: now 1781945693376385 us; error 31 us; skew 500 ppm
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:53.378978 3792 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:53.379808 3792 webserver.cc:460] Webserver started at http://127.0.98.3:38919/ using document root <none> and password file <none>
I20260620 08:54:53.380350 3792 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:53.380414 3792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:53.380571 3792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:53.382510 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "d11e318ac4ca43fa8c8daefec11f4360"
format_stamp: "Formatted at 2026-06-20 08:54:53 on dist-test-slave-fb1v"
server_key: "dc6c0c10e3f42c14680870ffdac45ca6"
server_key_iv: "eac65296e6589334c1075dcae3f6ac4f"
server_key_version: "encryptionkey@0"
I20260620 08:54:53.383013 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d11e318ac4ca43fa8c8daefec11f4360"
format_stamp: "Formatted at 2026-06-20 08:54:53 on dist-test-slave-fb1v"
server_key: "dc6c0c10e3f42c14680870ffdac45ca6"
server_key_iv: "eac65296e6589334c1075dcae3f6ac4f"
server_key_version: "encryptionkey@0"
I20260620 08:54:53.386869 3792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 08:54:53.389230 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:53.390477 3792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:53.390579 3792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "d11e318ac4ca43fa8c8daefec11f4360"
format_stamp: "Formatted at 2026-06-20 08:54:53 on dist-test-slave-fb1v"
server_key: "dc6c0c10e3f42c14680870ffdac45ca6"
server_key_iv: "eac65296e6589334c1075dcae3f6ac4f"
server_key_version: "encryptionkey@0"
I20260620 08:54:53.390689 3792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:53.415434 3792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:53.418519 3792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:53.418686 3792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:53.419246 3792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:53.420068 3792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:53.420114 3792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:53.420152 3792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:53.420167 3792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:53.428232 3792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35455
I20260620 08:54:53.428282 3920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35455 every 8 connection(s)
I20260620 08:54:53.429111 3792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:53.435444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3792
I20260620 08:54:53.435568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:53.435882 392 external_mini_cluster.cc:1428] Setting key f646263ac9de063e42225ad5f0ee768c
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:53.442770 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.430777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41473 (local address 127.0.98.62:34879)
0620 08:54:53.430917 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:53.430920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:53.431685 (+ 765us) server_negotiation.cc:407] Connection header received
0620 08:54:53.432983 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:53.432986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:53.433040 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:53.433112 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:53.434210 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.434725 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.436128 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.436431 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.436521 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.438914 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:53.438917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:53.438919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:53.438941 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:53.440220 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.441062 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.441065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.441066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.441146 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.441584 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.441587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.441588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.441761 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:53.441843 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:53.442421 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 08:54:53.442606 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:54:53.443502 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.431135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.3:41473)
0620 08:54:53.431538 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:53.431577 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:53.432797 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:53.433271 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:53.433277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:53.433622 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:53.434070 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:53.434082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.434836 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.434839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:53.436019 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:53.436030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.436582 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.436586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:53.436818 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.437434 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:53.437449 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:53.438776 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:53.440448 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:53.440456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:53.440475 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:53.440902 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:53.441299 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:53.441303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:53.441305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:53.441478 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:53.441859 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:53.441865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:53.442104 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 08:54:53.442281 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 08:54:53.442564 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260620 08:54:53.444733 3921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34879
I20260620 08:54:53.444983 3921 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:53.445431 3921 heartbeater.cc:511] Master 127.0.98.62:34879 requested a full tablet report, sending...
I20260620 08:54:53.446417 3462 ts_manager.cc:194] Registered new tserver with Master: d11e318ac4ca43fa8c8daefec11f4360 (127.0.98.3:35455)
I20260620 08:54:53.446956 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41473
I20260620 08:54:53.449918 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945693, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945693, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:53.474014 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:53.465804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.0.98.62:34879)
0620 08:54:53.465970 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:53.465975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:53.466090 (+ 115us) server_negotiation.cc:407] Connection header received
0620 08:54:53.466232 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:53.466235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:53.466281 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:53.466343 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:53.467039 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.467557 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.468350 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:53.468582 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:53.468667 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:53.471163 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:53.471166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:53.471168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:53.471192 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:53.472503 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.472863 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.472866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.472868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.472963 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:53.473193 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:53.473196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:53.473198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:53.473410 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:53.473527 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:53.473736 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 08:54:53.473844 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 08:54:53.474406 3462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53626
I20260620 08:54:53.475227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3524
I20260620 08:54:53.480065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3658
I20260620 08:54:53.484681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3792
I20260620 08:54:53.488924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3430
2026-06-20T08:54:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2043 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3933](info): set up 2 sockets
Jun 20 08:54:53 dist-test-slave-fb1v krb5kdc[3933](info): commencing operation
krb5kdc: starting...
W20260620 08:54:54.443435 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:54 dist-test-slave-fb1v krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:54.562726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:54.668732 3946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:54.668990 3946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:54.669044 3946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:54.673027 3946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:54.673084 3946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:54.673110 3946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:54.673133 3946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:54.673161 3946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:54.678361 3946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:54.679366 3946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:54.680346 3946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:54.688665 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:54.688846 3946 server_base.cc:1034] running on GCE node
W20260620 08:54:54.688644 3954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:54.688629 3951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:54.689467 3946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:54.690479 3946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:54.691656 3946 hybrid_clock.cc:648] HybridClock initialized: now 1781945694691643 us; error 26 us; skew 500 ppm
Jun 20 08:54:54 dist-test-slave-fb1v krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:54.694228 3946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:54.695019 3946 webserver.cc:460] Webserver started at http://127.0.98.62:35819/ using document root <none> and password file <none>
I20260620 08:54:54.695497 3946 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:54.695556 3946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:54.695762 3946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:54.697768 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "96f69924e21445299c760b0be1a638d7"
format_stamp: "Formatted at 2026-06-20 08:54:54 on dist-test-slave-fb1v"
server_key: "0880a6429df2a2c421dc443b6065bdd0"
server_key_iv: "26c8c7c225488d9257b5cbeb084e7383"
server_key_version: "encryptionkey@0"
I20260620 08:54:54.698268 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "96f69924e21445299c760b0be1a638d7"
format_stamp: "Formatted at 2026-06-20 08:54:54 on dist-test-slave-fb1v"
server_key: "0880a6429df2a2c421dc443b6065bdd0"
server_key_iv: "26c8c7c225488d9257b5cbeb084e7383"
server_key_version: "encryptionkey@0"
I20260620 08:54:54.701947 3946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:54.704300 3961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:54.705477 3946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:54.705575 3946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "96f69924e21445299c760b0be1a638d7"
format_stamp: "Formatted at 2026-06-20 08:54:54 on dist-test-slave-fb1v"
server_key: "0880a6429df2a2c421dc443b6065bdd0"
server_key_iv: "26c8c7c225488d9257b5cbeb084e7383"
server_key_version: "encryptionkey@0"
I20260620 08:54:54.705658 3946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:54.723395 3946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:54.726467 3946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:54.726619 3946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:54.732890 3946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37747
I20260620 08:54:54.732936 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37747 every 8 connection(s)
I20260620 08:54:54.733768 3946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:54.736402 4014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:54.739276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 3946
I20260620 08:54:54.739400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:54.739699 392 external_mini_cluster.cc:1428] Setting key 22aa8c68b7d888ee0bf66e114a4f97fa
I20260620 08:54:54.742002 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7: Bootstrap starting.
I20260620 08:54:54.744333 4014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:54.745008 4014 log.cc:824] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:54.746655 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7: No bootstrap required, opened a new log
Jun 20 08:54:54 dist-test-slave-fb1v krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:54.750118 4014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f69924e21445299c760b0be1a638d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260620 08:54:54.750346 4014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:54.750411 4014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f69924e21445299c760b0be1a638d7, State: Initialized, Role: FOLLOWER
I20260620 08:54:54.750828 4014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f69924e21445299c760b0be1a638d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260620 08:54:54.750954 4014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:54.751019 4014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:54.751111 4014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:54.752156 4014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f69924e21445299c760b0be1a638d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260620 08:54:54.752509 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96f69924e21445299c760b0be1a638d7; no voters:
I20260620 08:54:54.752837 4014 leader_election.cc:288] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:54.752926 4019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:54.753132 4019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [term 1 LEADER]: Becoming Leader. State: Replica: 96f69924e21445299c760b0be1a638d7, State: Running, Role: LEADER
I20260620 08:54:54.753454 4019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f69924e21445299c760b0be1a638d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260620 08:54:54.753854 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:54.740952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.0.98.62:37747)
0620 08:54:54.741332 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:54.741349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:54.741381 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:54:54.741964 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:54.741979 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:54.742324 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:54.742608 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:54.743434 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:54.744423 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:54.745238 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:54.745685 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:54.745824 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:54.748748 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:54.748754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:54.748766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:54.748794 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:54.750775 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:54.751269 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:54.751275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:54.751281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:54.751418 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:54.751662 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:54.751665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:54.751667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:54.752172 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:54.752367 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:54.752630 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 08:54:54.752955 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260620 08:54:54.754040 4014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:54.754549 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96f69924e21445299c760b0be1a638d7. Latest consensus state: current_term: 1 leader_uuid: "96f69924e21445299c760b0be1a638d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f69924e21445299c760b0be1a638d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } } }
I20260620 08:54:54.754683 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:54.754968 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96f69924e21445299c760b0be1a638d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f69924e21445299c760b0be1a638d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } } }
I20260620 08:54:54.755054 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:54.756002 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:54.758682 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:54.763733 4025 catalog_manager.cc:1347] Generated new cluster ID: 1304df40d90a498fa1d74d8520218918
I20260620 08:54:54.763782 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:54.777506 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:54.778355 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:54.795354 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96f69924e21445299c760b0be1a638d7: Generated new TSK 0
I20260620 08:54:54.796132 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:54.824911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:54.932334 4040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:54.932581 4040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:54.932622 4040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:54.936559 4040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:54.936609 4040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:54.936667 4040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:54.941759 4040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:54.942740 4040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:54.943696 4040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:54.952731 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:54.952731 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:54.952863 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:54.952942 4040 server_base.cc:1034] running on GCE node
I20260620 08:54:54.953351 4040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:54.953989 4040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:54.955096 4040 hybrid_clock.cc:648] HybridClock initialized: now 1781945694955076 us; error 35 us; skew 500 ppm
Jun 20 08:54:54 dist-test-slave-fb1v krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:54.957708 4040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:54.958516 4040 webserver.cc:460] Webserver started at http://127.0.98.1:38447/ using document root <none> and password file <none>
I20260620 08:54:54.959019 4040 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:54.959075 4040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:54.959239 4040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:54.961191 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "11471179948847f6bc7c45b9f9dc5278"
format_stamp: "Formatted at 2026-06-20 08:54:54 on dist-test-slave-fb1v"
server_key: "d27c81badd7bac67f452c7a039098a77"
server_key_iv: "1caa8996e243e56185797779b2aed446"
server_key_version: "encryptionkey@0"
I20260620 08:54:54.961658 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "11471179948847f6bc7c45b9f9dc5278"
format_stamp: "Formatted at 2026-06-20 08:54:54 on dist-test-slave-fb1v"
server_key: "d27c81badd7bac67f452c7a039098a77"
server_key_iv: "1caa8996e243e56185797779b2aed446"
server_key_version: "encryptionkey@0"
I20260620 08:54:54.965095 4040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:54.967444 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:54.968770 4040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:54.968874 4040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "11471179948847f6bc7c45b9f9dc5278"
format_stamp: "Formatted at 2026-06-20 08:54:54 on dist-test-slave-fb1v"
server_key: "d27c81badd7bac67f452c7a039098a77"
server_key_iv: "1caa8996e243e56185797779b2aed446"
server_key_version: "encryptionkey@0"
I20260620 08:54:54.968983 4040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:54.997390 4040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:55.000957 4040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:55.001116 4040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:55.001677 4040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:55.002472 4040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:55.002532 4040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.002571 4040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:55.002592 4040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.012185 4040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40085
I20260620 08:54:55.012215 4168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40085 every 8 connection(s)
I20260620 08:54:55.013051 4040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:55.021257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4040
I20260620 08:54:55.021391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:55.021824 392 external_mini_cluster.cc:1428] Setting key f856ab90f751864dde78ed8a1323a05d
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:55.028173 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.015170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40697 (local address 127.0.98.62:37747)
0620 08:54:55.015342 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:55.015346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:55.016222 (+ 876us) server_negotiation.cc:407] Connection header received
0620 08:54:55.017563 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:55.017566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:55.017620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:55.017696 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:55.018828 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.019372 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.020533 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.020764 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.020852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.024000 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:55.024005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:55.024008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:55.024036 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:55.025870 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.026658 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.026661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.026662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.026741 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.027081 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.027085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.027086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.027260 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:55.027351 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:55.027879 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 08:54:55.028011 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 08:54:55.028978 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.015535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.1:40697)
0620 08:54:55.016063 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:55.016105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:55.017365 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:55.017856 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:55.017863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:55.018215 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:55.018683 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:55.018696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.019490 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.019493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:55.020417 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:55.020428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.020898 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.020903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:55.021093 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.021967 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:55.021998 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:55.023837 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:55.026028 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:55.026036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:55.026052 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:55.026545 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:55.026825 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:55.026828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:55.026830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:55.026986 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:55.027371 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:55.027378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:55.027640 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 08:54:55.027815 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 08:54:55.028172 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":119,"threads_started":1}
I20260620 08:54:55.030308 4169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260620 08:54:55.030532 4169 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:54:55.030995 4169 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260620 08:54:55.032888 3978 ts_manager.cc:194] Registered new tserver with Master: 11471179948847f6bc7c45b9f9dc5278 (127.0.98.1:40085)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:54:55.034225 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:55.050227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:55.156468 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:55.156701 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:55.156741 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:55.160998 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:55.161072 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:55.161134 4174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:55.167202 4174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:55.168213 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:55.169142 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:55.178242 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:55.178270 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:55.178454 4174 server_base.cc:1034] running on GCE node
W20260620 08:54:55.178251 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:55.178905 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:55.179463 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:55.180629 4174 hybrid_clock.cc:648] HybridClock initialized: now 1781945695180617 us; error 45 us; skew 500 ppm
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:55.183077 4174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:55.183928 4174 webserver.cc:460] Webserver started at http://127.0.98.2:39499/ using document root <none> and password file <none>
I20260620 08:54:55.184432 4174 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:55.184492 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:55.184670 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:55.186548 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "69f01a6c7bb24890b7b08fc7c0a537c7"
format_stamp: "Formatted at 2026-06-20 08:54:55 on dist-test-slave-fb1v"
server_key: "5e4b9694fb8a01443d91bbc5387cf82a"
server_key_iv: "13e6656ea9dbb200b7854eb44013de55"
server_key_version: "encryptionkey@0"
I20260620 08:54:55.187033 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "69f01a6c7bb24890b7b08fc7c0a537c7"
format_stamp: "Formatted at 2026-06-20 08:54:55 on dist-test-slave-fb1v"
server_key: "5e4b9694fb8a01443d91bbc5387cf82a"
server_key_iv: "13e6656ea9dbb200b7854eb44013de55"
server_key_version: "encryptionkey@0"
I20260620 08:54:55.190428 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:54:55.192579 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.193660 4174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:55.193755 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "69f01a6c7bb24890b7b08fc7c0a537c7"
format_stamp: "Formatted at 2026-06-20 08:54:55 on dist-test-slave-fb1v"
server_key: "5e4b9694fb8a01443d91bbc5387cf82a"
server_key_iv: "13e6656ea9dbb200b7854eb44013de55"
server_key_version: "encryptionkey@0"
I20260620 08:54:55.193862 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:55.217119 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:55.219740 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:55.219911 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:55.220449 4174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:55.221246 4174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:55.221290 4174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.221340 4174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:55.221369 4174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.229858 4174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33901
I20260620 08:54:55.229878 4302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33901 every 8 connection(s)
I20260620 08:54:55.230669 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:55.236214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4174
I20260620 08:54:55.236339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:55.236652 392 external_mini_cluster.cc:1428] Setting key 7461bcbed1a02b6e17bb91ef1256d200
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:55.244380 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.232358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35647 (local address 127.0.98.62:37747)
0620 08:54:55.232472 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:55.232477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:55.233240 (+ 763us) server_negotiation.cc:407] Connection header received
0620 08:54:55.234559 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:55.234562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:55.234617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:55.234690 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:55.235842 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.236384 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.237748 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.238008 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.238088 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.240953 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:55.240956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:55.240958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:55.240981 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:55.242311 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.242914 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.242917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.242918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.242992 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.243332 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.243334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.243335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.243495 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:55.243563 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:55.244168 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 08:54:55.244265 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:54:55.245479 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.232694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.2:35647)
0620 08:54:55.233108 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:55.233147 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:55.234361 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:55.234854 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:55.234860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:55.235192 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:55.235714 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:55.235729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.236505 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.236509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:55.237628 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:55.237639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.238153 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.238158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:55.238346 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.238909 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:55.238929 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:55.240788 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:55.242422 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:55.242429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:55.242444 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:55.242807 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:55.243074 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:55.243076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:55.243079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:55.243249 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:55.243593 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:55.243599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:55.243933 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 08:54:55.244142 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 08:54:55.244470 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:54:55.246533 4303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260620 08:54:55.246747 4303 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:55.247212 4303 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:54:55.248205 3978 ts_manager.cc:194] Registered new tserver with Master: 69f01a6c7bb24890b7b08fc7c0a537c7 (127.0.98.2:33901)
I20260620 08:54:55.248878 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:55.262624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:55.373301 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:55.373550 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:55.373625 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:55.377593 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:55.377642 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:55.377718 4308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:55.382704 4308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:55.383653 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:55.384594 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:55.393930 4313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:55.393911 4314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:55.393869 4316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:55.394215 4308 server_base.cc:1034] running on GCE node
I20260620 08:54:55.394603 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:55.395170 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:55.396355 4308 hybrid_clock.cc:648] HybridClock initialized: now 1781945695396328 us; error 36 us; skew 500 ppm
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:55.398931 4308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:55.399767 4308 webserver.cc:460] Webserver started at http://127.0.98.3:37061/ using document root <none> and password file <none>
I20260620 08:54:55.400254 4308 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:55.400306 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:55.400458 4308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:55.402297 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "57ada7b50fd64f369d9b0e34633d2dc4"
format_stamp: "Formatted at 2026-06-20 08:54:55 on dist-test-slave-fb1v"
server_key: "fa00fd00c5a3813ae78c0c325cb6aa49"
server_key_iv: "4227bcc9653920acabe5f2a94b688f2e"
server_key_version: "encryptionkey@0"
I20260620 08:54:55.402784 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57ada7b50fd64f369d9b0e34633d2dc4"
format_stamp: "Formatted at 2026-06-20 08:54:55 on dist-test-slave-fb1v"
server_key: "fa00fd00c5a3813ae78c0c325cb6aa49"
server_key_iv: "4227bcc9653920acabe5f2a94b688f2e"
server_key_version: "encryptionkey@0"
I20260620 08:54:55.406173 4308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:54:55.408407 4323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.409572 4308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:55.409672 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "57ada7b50fd64f369d9b0e34633d2dc4"
format_stamp: "Formatted at 2026-06-20 08:54:55 on dist-test-slave-fb1v"
server_key: "fa00fd00c5a3813ae78c0c325cb6aa49"
server_key_iv: "4227bcc9653920acabe5f2a94b688f2e"
server_key_version: "encryptionkey@0"
I20260620 08:54:55.409759 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:55.439868 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:55.442760 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:55.442935 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:55.443483 4308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:55.444283 4308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:55.444329 4308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.444379 4308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:55.444407 4308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:55.452402 4308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43509
I20260620 08:54:55.452430 4436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43509 every 8 connection(s)
I20260620 08:54:55.453212 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:55.458495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4308
I20260620 08:54:55.458629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:55.458878 392 external_mini_cluster.cc:1428] Setting key d02ad72aef89ab10cda62618769c8063
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:55.466562 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.454900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58467 (local address 127.0.98.62:37747)
0620 08:54:55.455037 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:55.455042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:55.455912 (+ 870us) server_negotiation.cc:407] Connection header received
0620 08:54:55.457326 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:55.457330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:55.457389 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:55.457470 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:55.458622 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.459263 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.460692 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.461009 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.461099 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.463516 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:55.463519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:55.463521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:55.463545 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:55.464823 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.465259 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.465262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.465263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.465347 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.465641 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.465643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.465645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.465819 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:55.465905 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:55.466333 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 08:54:55.466441 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:54:55.467303 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.455274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.3:58467)
0620 08:54:55.455746 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:55.455796 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:55.457109 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:55.457645 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:55.457652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:55.458011 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:55.458463 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:55.458477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.459385 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.459389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:55.460567 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:55.460577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.461161 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.461165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:55.461327 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.461910 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:55.461927 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:55.463361 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:55.464928 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:55.464932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:55.464942 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:55.465170 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:55.465414 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:55.465416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:55.465417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:55.465542 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:55.465912 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:55.465917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:55.466131 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 08:54:55.466286 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 08:54:55.466531 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260620 08:54:55.468439 4437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260620 08:54:55.468652 4437 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:55.469108 4437 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260620 08:54:55.470006 3978 ts_manager.cc:194] Registered new tserver with Master: 57ada7b50fd64f369d9b0e34633d2dc4 (127.0.98.3:43509)
I20260620 08:54:55.470527 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58467
I20260620 08:54:55.472656 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:54:55.496287 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.487338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.0.98.1:40085)
0620 08:54:55.487635 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:55.487650 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:55.487715 (+ 65us) server_negotiation.cc:407] Connection header received
0620 08:54:55.487844 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:55.487850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:55.488041 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:55.488168 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:55.488792 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.489549 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.490340 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.490693 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.490786 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.492978 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:55.492986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:55.492997 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:55.493018 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:55.494415 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.494876 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.494881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.494882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.494976 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.495236 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.495240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.495242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.495492 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:55.495734 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:55.495941 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 08:54:55.496091 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:55.506057 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:55.498895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.98.62:37747)
0620 08:54:55.499009 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:55.499012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:55.499042 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:54:55.499139 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:55.499141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:55.499177 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:55.499248 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:55.499815 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.500308 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.501060 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:55.501265 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:55.501340 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:55.503420 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:55.503423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:55.503425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:55.503447 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:55.504636 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.504992 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.504996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.504997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.505062 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:55.505284 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:55.505288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:55.505289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:55.505484 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:55.505596 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:55.505802 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:54:55.505905 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 08:54:55.506410 3978 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56432
I20260620 08:54:55.506958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4040
I20260620 08:54:55.512460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4174
I20260620 08:54:55.516880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4308
I20260620 08:54:55.521047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 3946
2026-06-20T08:54:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2032 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[4449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[4449](info): set up 2 sockets
Jun 20 08:54:55 dist-test-slave-fb1v krb5kdc[4449](info): commencing operation
krb5kdc: starting...
W20260620 08:54:56.479394 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:54:56 dist-test-slave-fb1v krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:54:56.591216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:56.700008 4462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:56.700276 4462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:56.700335 4462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:56.704447 4462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:56.704514 4462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:56.704541 4462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:56.704563 4462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:56.704592 4462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:56.709892 4462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:56.710883 4462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:56.711897 4462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:56.720530 4468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:56.720762 4462 server_base.cc:1034] running on GCE node
W20260620 08:54:56.720530 4470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:56.720572 4467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:56.721840 4462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:56.723048 4462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:56.724208 4462 hybrid_clock.cc:648] HybridClock initialized: now 1781945696724184 us; error 38 us; skew 500 ppm
Jun 20 08:54:56 dist-test-slave-fb1v krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:56.726900 4462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:54:56.727671 4462 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:56.727747 4462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:56.727950 4462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:56.729977 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "be1417e485024767b335eb1e9e9a0153"
format_stamp: "Formatted at 2026-06-20 08:54:56 on dist-test-slave-fb1v"
server_key: "b74c8a16ba5714cff011296ec54acde5"
server_key_iv: "a4caf27cd3fdc49212b7d50bd3e0f36a"
server_key_version: "encryptionkey@0"
I20260620 08:54:56.730497 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "be1417e485024767b335eb1e9e9a0153"
format_stamp: "Formatted at 2026-06-20 08:54:56 on dist-test-slave-fb1v"
server_key: "b74c8a16ba5714cff011296ec54acde5"
server_key_iv: "a4caf27cd3fdc49212b7d50bd3e0f36a"
server_key_version: "encryptionkey@0"
I20260620 08:54:56.734094 4462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:54:56.736388 4476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:56.737550 4462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:56.737663 4462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "be1417e485024767b335eb1e9e9a0153"
format_stamp: "Formatted at 2026-06-20 08:54:56 on dist-test-slave-fb1v"
server_key: "b74c8a16ba5714cff011296ec54acde5"
server_key_iv: "a4caf27cd3fdc49212b7d50bd3e0f36a"
server_key_version: "encryptionkey@0"
I20260620 08:54:56.737751 4462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:56.753438 4462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:56.756479 4462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:56.756651 4462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:56.763171 4462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34395
I20260620 08:54:56.763407 4528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34395 every 8 connection(s)
I20260620 08:54:56.763821 4462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:56.766366 4529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:56.767027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4462
I20260620 08:54:56.767149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:56.767433 392 external_mini_cluster.cc:1428] Setting key 9d66a03c907d3ee5da3b0344ef60e7cf
I20260620 08:54:56.772115 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153: Bootstrap starting.
I20260620 08:54:56.774186 4529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:56.774861 4529 log.cc:824] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:54:56 dist-test-slave-fb1v krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:56.776866 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153: No bootstrap required, opened a new log
I20260620 08:54:56.780686 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:56.768801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.98.62:34395)
0620 08:54:56.769210 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:56.769225 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:56.769258 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:54:56.769886 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:56.769901 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:56.770221 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:56.770537 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:56.771274 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:56.772318 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:56.773125 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:56.773607 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:56.773748 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:56.776244 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:56.776249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:56.776261 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:56.776287 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:56.777895 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:56.778377 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:56.778382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:56.778386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:56.778505 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:56.778785 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:56.778788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:56.778789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:56.779156 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:56.779291 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:56.779561 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 08:54:56.779934 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
I20260620 08:54:56.781279 4529 raft_consensus.cc:348] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1417e485024767b335eb1e9e9a0153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34395 } }
I20260620 08:54:56.781497 4529 raft_consensus.cc:374] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:56.781535 4529 raft_consensus.cc:729] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1417e485024767b335eb1e9e9a0153, State: Initialized, Role: FOLLOWER
I20260620 08:54:56.782002 4529 consensus_queue.cc:260] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1417e485024767b335eb1e9e9a0153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34395 } }
I20260620 08:54:56.782162 4529 raft_consensus.cc:388] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:56.782236 4529 raft_consensus.cc:482] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:56.782322 4529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:56.783265 4529 raft_consensus.cc:504] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1417e485024767b335eb1e9e9a0153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34395 } }
I20260620 08:54:56.783540 4529 leader_election.cc:302] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be1417e485024767b335eb1e9e9a0153; no voters:
I20260620 08:54:56.783818 4529 leader_election.cc:288] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:56.783905 4534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:56.784119 4534 raft_consensus.cc:686] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [term 1 LEADER]: Becoming Leader. State: Replica: be1417e485024767b335eb1e9e9a0153, State: Running, Role: LEADER
I20260620 08:54:56.784402 4534 consensus_queue.cc:237] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1417e485024767b335eb1e9e9a0153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34395 } }
I20260620 08:54:56.784947 4529 sys_catalog.cc:564] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:56.786026 4535 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be1417e485024767b335eb1e9e9a0153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1417e485024767b335eb1e9e9a0153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34395 } } }
I20260620 08:54:56.786144 4535 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:56.786561 4536 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1417e485024767b335eb1e9e9a0153. Latest consensus state: current_term: 1 leader_uuid: "be1417e485024767b335eb1e9e9a0153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1417e485024767b335eb1e9e9a0153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34395 } } }
I20260620 08:54:56.787117 4536 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:56.787192 4544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:56.789731 4544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:56.794858 4544 catalog_manager.cc:1347] Generated new cluster ID: f5fce4a004d948c98371ae191212d029
I20260620 08:54:56.794921 4544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:56.805475 4544 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:56.806653 4544 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:56.815718 4544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153: Generated new TSK 0
I20260620 08:54:56.816366 4544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:54:56.850811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34395
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:56.959386 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:56.959678 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:56.959743 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:56.963750 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:56.963807 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:56.963886 4555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:56.969144 4555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:56.970130 4555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:56.971091 4555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:56.979947 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:56.979996 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:56.979967 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:56.980203 4555 server_base.cc:1034] running on GCE node
I20260620 08:54:56.980638 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:56.981258 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:56.982419 4555 hybrid_clock.cc:648] HybridClock initialized: now 1781945696982401 us; error 34 us; skew 500 ppm
Jun 20 08:54:56 dist-test-slave-fb1v krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:56.985262 4555 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:54:56.986032 4555 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:56.986095 4555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:56.986294 4555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:56.988381 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "b75703e5624545ee8110fbe964699b9c"
format_stamp: "Formatted at 2026-06-20 08:54:56 on dist-test-slave-fb1v"
server_key: "e0bdd923f91c961e633bed18b5ef028a"
server_key_iv: "9e7d4d1c7575a5ef831fba9347226a2e"
server_key_version: "encryptionkey@0"
I20260620 08:54:56.988903 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b75703e5624545ee8110fbe964699b9c"
format_stamp: "Formatted at 2026-06-20 08:54:56 on dist-test-slave-fb1v"
server_key: "e0bdd923f91c961e633bed18b5ef028a"
server_key_iv: "9e7d4d1c7575a5ef831fba9347226a2e"
server_key_version: "encryptionkey@0"
I20260620 08:54:56.992424 4555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:54:56.994920 4569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:56.996201 4555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:56.996320 4555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "b75703e5624545ee8110fbe964699b9c"
format_stamp: "Formatted at 2026-06-20 08:54:56 on dist-test-slave-fb1v"
server_key: "e0bdd923f91c961e633bed18b5ef028a"
server_key_iv: "9e7d4d1c7575a5ef831fba9347226a2e"
server_key_version: "encryptionkey@0"
I20260620 08:54:56.996423 4555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:57.018615 4555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:57.021651 4555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:57.021827 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:57.022388 4555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:57.023223 4555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:57.023273 4555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.023326 4555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:57.023352 4555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.031901 4555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32973
I20260620 08:54:57.031936 4682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32973 every 8 connection(s)
I20260620 08:54:57.032454 4555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:57.036710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4555
I20260620 08:54:57.036820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:57.037109 392 external_mini_cluster.cc:1428] Setting key ca97f309d336bc344911c7329fc528a0
Jun 20 08:54:57 dist-test-slave-fb1v krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:54:57.047114 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.034134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52565 (local address 127.0.98.62:34395)
0620 08:54:57.034278 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.034293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.035419 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 08:54:57.036921 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.036926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.036994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.037087 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:57.038490 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.039053 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.040463 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.040701 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.040779 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.043571 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:57.043574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:57.043576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:57.043597 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:57.044889 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.045511 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.045514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.045515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.045599 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.045997 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.046000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.046002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.046192 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:57.046283 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.046886 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.046988 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:54:57.047907 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.034455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34395 (local address 127.0.98.1:52565)
0620 08:54:57.035175 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.035248 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.036693 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.037285 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.037292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.037738 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:57.038331 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.038349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.039185 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.039188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.040335 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.040347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.040870 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.040875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.041072 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.041673 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:57.041691 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:57.043399 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:57.045015 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:57.045022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:57.045037 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:57.045395 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:57.045692 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:57.045696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:57.045698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:57.045872 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:57.046300 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:57.046306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:57.046635 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.046816 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.047071 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:54:57.049135 4683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34395
I20260620 08:54:57.049357 4683 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:57.049779 4683 heartbeater.cc:511] Master 127.0.98.62:34395 requested a full tablet report, sending...
I20260620 08:54:57.051175 4491 ts_manager.cc:194] Registered new tserver with Master: b75703e5624545ee8110fbe964699b9c (127.0.98.1:32973)
I20260620 08:54:57.052296 4491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:54:57.062916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34395
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:57.168632 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:57.168885 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:57.168941 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:57.173183 4688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:57.173241 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:57.173322 4688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:57.178643 4688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:57.179570 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:57.180546 4688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:57.189976 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:57.190021 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:57.190016 4688 server_base.cc:1034] running on GCE node
W20260620 08:54:57.189976 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:57.190603 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:57.191205 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:57.192348 4688 hybrid_clock.cc:648] HybridClock initialized: now 1781945697192331 us; error 30 us; skew 500 ppm
Jun 20 08:54:57 dist-test-slave-fb1v krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:57.194784 4688 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:54:57.195443 4688 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:57.195503 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:57.195698 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:57.197557 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ed0743c3ba0455290d35484ad0a3f3c"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "5547c8f0fc9b6331074d106ec4c368bd"
server_key_iv: "27b41cfaba353f39a364120d1c1d940a"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.198027 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ed0743c3ba0455290d35484ad0a3f3c"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "5547c8f0fc9b6331074d106ec4c368bd"
server_key_iv: "27b41cfaba353f39a364120d1c1d940a"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.201649 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:57.203804 4702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.204927 4688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:57.205030 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "2ed0743c3ba0455290d35484ad0a3f3c"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "5547c8f0fc9b6331074d106ec4c368bd"
server_key_iv: "27b41cfaba353f39a364120d1c1d940a"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.205124 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:57.219477 4688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:57.222597 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:57.222769 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:57.223321 4688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:57.224166 4688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:57.224215 4688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.224252 4688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:57.224267 4688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.232463 4688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41569
I20260620 08:54:57.232483 4815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41569 every 8 connection(s)
I20260620 08:54:57.233001 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:57.238305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4688
I20260620 08:54:57.238405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:57.238684 392 external_mini_cluster.cc:1428] Setting key 7f6de2dad6b1491b2d673a44eee94297
Jun 20 08:54:57 dist-test-slave-fb1v krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:57.246917 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.234570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48205 (local address 127.0.98.62:34395)
0620 08:54:57.234694 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.234698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.235453 (+ 755us) server_negotiation.cc:407] Connection header received
0620 08:54:57.236849 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.236852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.236907 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.236976 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:57.238103 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.238665 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.239865 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.240145 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.240253 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.243406 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:57.243409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:57.243411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:57.243435 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:57.244754 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.245345 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.245348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.245350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.245431 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.245801 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.245804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.245806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.245992 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:57.246070 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.246681 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.246788 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:54:57.248003 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.234906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34395 (local address 127.0.98.2:48205)
0620 08:54:57.235318 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.235358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.236652 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.237143 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.237150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.237522 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:57.237965 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.237978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.238785 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.238788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.239756 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.239765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.240281 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.240284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.240419 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.241536 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:57.241558 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:57.243225 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:57.244864 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:57.244870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:57.244886 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:57.245232 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:57.245532 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:57.245535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:57.245537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:57.245706 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:57.246093 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:57.246100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:57.246407 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.246626 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.246962 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20260620 08:54:57.249457 4816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34395
I20260620 08:54:57.249707 4816 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:54:57.250185 4816 heartbeater.cc:511] Master 127.0.98.62:34395 requested a full tablet report, sending...
I20260620 08:54:57.251477 4491 ts_manager.cc:194] Registered new tserver with Master: 2ed0743c3ba0455290d35484ad0a3f3c (127.0.98.2:41569)
I20260620 08:54:57.252125 4491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:54:57.267314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34395
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:54:57.378235 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:57.378475 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:57.378520 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:57.382454 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:57.382507 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:57.382581 4821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:57.387723 4821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:57.388602 4821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:57.389478 4821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:57.398145 4827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:57.398147 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:57.398195 4821 server_base.cc:1034] running on GCE node
W20260620 08:54:57.398145 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:57.398706 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:57.399281 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:57.400437 4821 hybrid_clock.cc:648] HybridClock initialized: now 1781945697400414 us; error 35 us; skew 500 ppm
Jun 20 08:54:57 dist-test-slave-fb1v krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:54:57.403172 4821 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:54:57.403923 4821 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:57.403990 4821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:57.404167 4821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:57.406088 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "24ac612e4bde4f8eacb49dcf0444e669"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "bb788558dde4d822faa1a0ce5c8daedc"
server_key_iv: "8a5146786074be2f4b82cad6413f30a0"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.406612 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "24ac612e4bde4f8eacb49dcf0444e669"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "bb788558dde4d822faa1a0ce5c8daedc"
server_key_iv: "8a5146786074be2f4b82cad6413f30a0"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.410333 4821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:54:57.412694 4835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.414069 4821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:54:57.414182 4821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "24ac612e4bde4f8eacb49dcf0444e669"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "bb788558dde4d822faa1a0ce5c8daedc"
server_key_iv: "8a5146786074be2f4b82cad6413f30a0"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.414281 4821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:57.428323 4821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:57.431260 4821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:57.431432 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:57.431989 4821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:57.432771 4821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:57.432818 4821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.432868 4821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:57.432896 4821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.440514 4821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35097
I20260620 08:54:57.440543 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35097 every 8 connection(s)
I20260620 08:54:57.441104 4821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:57.443527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4821
I20260620 08:54:57.443665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:57.443970 392 external_mini_cluster.cc:1428] Setting key 9152af72f7cef208d08b8ae476a784f6
Jun 20 08:54:57 dist-test-slave-fb1v krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:54:57.454892 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.442884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44145 (local address 127.0.98.62:34395)
0620 08:54:57.443039 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.443043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.443936 (+ 893us) server_negotiation.cc:407] Connection header received
0620 08:54:57.445296 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.445299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.445347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.445458 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:57.446679 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.447199 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.448404 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.448645 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.448737 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.451455 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:57.451459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:57.451461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:57.451486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:57.452936 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.453473 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.453476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.453477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.453559 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.453894 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.453896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.453897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.454074 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:57.454151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.454653 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.454766 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:54:57.455698 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.443233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34395 (local address 127.0.98.3:44145)
0620 08:54:57.443758 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.443815 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.445094 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.445626 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.445632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.446013 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:57.446528 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.446544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.447327 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.447330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.448282 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.448293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.448796 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.448799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.448971 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.449549 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:54:57.449565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:57.451243 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:57.453067 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:57.453074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:57.453086 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:57.453378 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:57.453662 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:54:57.453664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:54:57.453666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:54:57.453808 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:54:57.454171 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:57.454177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:57.454413 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.454582 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.454844 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":110,"threads_started":1}
I20260620 08:54:57.456830 4949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34395
I20260620 08:54:57.457075 4949 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:57.457572 4949 heartbeater.cc:511] Master 127.0.98.62:34395 requested a full tablet report, sending...
I20260620 08:54:57.458817 4491 ts_manager.cc:194] Registered new tserver with Master: 24ac612e4bde4f8eacb49dcf0444e669 (127.0.98.3:35097)
I20260620 08:54:57.459563 4491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44145
I20260620 08:54:57.469640 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:54:57.478098 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.471413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.98.62:34395)
0620 08:54:57.471545 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.471550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.471707 (+ 157us) server_negotiation.cc:407] Connection header received
0620 08:54:57.471814 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.471817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.471857 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.471921 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:57.472580 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.473074 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.473847 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.474062 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.474146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.475440 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:57.475443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:57.475444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:54:57.475465 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:57.476599 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.476951 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.476955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.476957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.477027 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:54:57.477251 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:54:57.477254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:54:57.477256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:54:57.477483 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:57.477640 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.477835 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.477943 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:54:57.481096 4491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:54:57.484083 4491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:57.498421 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.493033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35097 (local address 127.0.0.1:56870)
0620 08:54:57.493851 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.493862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.493930 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.494529 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.494531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.494554 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:57.494684 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.494690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.495690 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.495694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.496902 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.496911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.497908 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.497911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.498012 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.498051 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.498157 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.498264 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":84,"spinlock_wait_cycles":1792,"thread_start_us":97,"threads_started":1}
I20260620 08:54:57.498453 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.493376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.0.98.3:35097)
0620 08:54:57.494147 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.494158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.494178 (+ 20us) server_negotiation.cc:407] Connection header received
0620 08:54:57.494245 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.494250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.494406 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.494536 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:57.494805 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.495545 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.497023 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.497802 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.497911 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.498049 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.498197 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.498304 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":133,"threads_started":1}
I20260620 08:54:57.498770 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.492713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.0.98.1:32973)
0620 08:54:57.493130 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.493142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.493610 (+ 468us) server_negotiation.cc:407] Connection header received
0620 08:54:57.493959 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.493966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.494168 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.494433 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:57.494830 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.495550 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.497118 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.497824 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.497911 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.498049 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.498550 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.498639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":106,"threads_started":1}
I20260620 08:54:57.498466 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.492568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32973 (local address 127.0.0.1:40768)
0620 08:54:57.493480 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.493531 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.493684 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.494581 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.494584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.494599 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:57.494700 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.494705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.495684 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.495687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.497015 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.497022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.498110 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.498113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.498206 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.498228 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.498293 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.498353 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":69,"thread_start_us":118,"threads_started":1}
I20260620 08:54:57.499182 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.492678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41569 (local address 127.0.0.1:52568)
0620 08:54:57.493479 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.493530 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.493650 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.494488 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.494503 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.494525 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:57.494684 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.494690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.496988 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.496991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.497711 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.497719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.498827 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.498830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.498913 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.498934 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.498999 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.499065 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":30,"thread_start_us":107,"threads_started":1}
I20260620 08:54:57.499567 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.493379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.98.2:41569)
0620 08:54:57.493897 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.493907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.493928 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:54:57.493998 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.494003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.494192 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.494335 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:57.495704 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.496846 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.497822 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.498559 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.498652 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.498769 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.499339 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.499429 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":110,"threads_started":1}
I20260620 08:54:57.502000 4883 tablet_service.cc:1467] Processing CreateTablet for tablet 389fcee462a6448286ba80071cf6c11a (DEFAULT_TABLE table=test-table [id=7927c22b8d404d84a3c17c2df92360ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:57.502187 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 389fcee462a6448286ba80071cf6c11a (DEFAULT_TABLE table=test-table [id=7927c22b8d404d84a3c17c2df92360ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:57.502269 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 389fcee462a6448286ba80071cf6c11a (DEFAULT_TABLE table=test-table [id=7927c22b8d404d84a3c17c2df92360ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:57.503100 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389fcee462a6448286ba80071cf6c11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:57.503104 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389fcee462a6448286ba80071cf6c11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:57.503113 4883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389fcee462a6448286ba80071cf6c11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:57.509032 4966 tablet_bootstrap.cc:492] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: Bootstrap starting.
I20260620 08:54:57.510423 4967 tablet_bootstrap.cc:492] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Bootstrap starting.
I20260620 08:54:57.511137 4966 tablet_bootstrap.cc:654] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:57.511924 4968 tablet_bootstrap.cc:492] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Bootstrap starting.
I20260620 08:54:57.512156 4967 tablet_bootstrap.cc:654] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:57.512180 4966 log.cc:824] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:57.513304 4967 log.cc:824] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:57.513887 4968 tablet_bootstrap.cc:654] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:57.514034 4966 tablet_bootstrap.cc:492] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: No bootstrap required, opened a new log
I20260620 08:54:57.514253 4966 ts_tablet_manager.cc:1397] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:54:57.514914 4968 log.cc:824] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:57.515228 4967 tablet_bootstrap.cc:492] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: No bootstrap required, opened a new log
I20260620 08:54:57.515434 4967 ts_tablet_manager.cc:1397] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:54:57.516392 4968 tablet_bootstrap.cc:492] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: No bootstrap required, opened a new log
I20260620 08:54:57.516551 4968 ts_tablet_manager.cc:1397] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:54:57.517947 4966 raft_consensus.cc:348] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.518215 4966 raft_consensus.cc:374] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:57.518301 4966 raft_consensus.cc:729] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75703e5624545ee8110fbe964699b9c, State: Initialized, Role: FOLLOWER
I20260620 08:54:57.518741 4966 consensus_queue.cc:260] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.519381 4683 heartbeater.cc:503] Master 127.0.98.62:34395 was elected leader, sending a full tablet report...
I20260620 08:54:57.519714 4966 ts_tablet_manager.cc:1428] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:54:57.519651 4968 raft_consensus.cc:348] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.519883 4968 raft_consensus.cc:374] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:57.519935 4968 raft_consensus.cc:729] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed0743c3ba0455290d35484ad0a3f3c, State: Initialized, Role: FOLLOWER
I20260620 08:54:57.520284 4968 consensus_queue.cc:260] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.519183 4967 raft_consensus.cc:348] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.520506 4967 raft_consensus.cc:374] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:57.520979 4967 raft_consensus.cc:729] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ac612e4bde4f8eacb49dcf0444e669, State: Initialized, Role: FOLLOWER
I20260620 08:54:57.521252 4968 ts_tablet_manager.cc:1428] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:54:57.521377 4967 consensus_queue.cc:260] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.521519 4816 heartbeater.cc:503] Master 127.0.98.62:34395 was elected leader, sending a full tablet report...
I20260620 08:54:57.522251 4967 ts_tablet_manager.cc:1428] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260620 08:54:57.522327 4949 heartbeater.cc:503] Master 127.0.98.62:34395 was elected leader, sending a full tablet report...
W20260620 08:54:57.533442 4684 tablet.cc:2369] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:57.577666 4974 raft_consensus.cc:482] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:57.577837 4974 raft_consensus.cc:504] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.578894 4974 leader_election.cc:288] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75703e5624545ee8110fbe964699b9c (127.0.98.1:32973), 2ed0743c3ba0455290d35484ad0a3f3c (127.0.98.2:41569)
I20260620 08:54:57.582229 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.579148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32973 (local address 127.0.98.3:40625)
0620 08:54:57.579241 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.579254 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.579332 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.579584 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.579588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.579633 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:57.579739 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.579745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.580426 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.580429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.581211 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.581220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.581832 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.581835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.581937 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.581954 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.582025 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.582097 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 08:54:57.582260 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.579204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40625 (local address 127.0.98.1:32973)
0620 08:54:57.579316 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.579320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.579337 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:54:57.579427 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.579430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.579476 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.579557 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:57.579842 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.580327 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.581329 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.581728 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.581819 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.581879 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.582049 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.582112 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:54:57.582701 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.579305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45489 (local address 127.0.98.2:41569)
0620 08:54:57.579442 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.579446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.579850 (+ 404us) server_negotiation.cc:407] Connection header received
0620 08:54:57.579956 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.579959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.580008 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.580065 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:54:57.580296 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.580777 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.581773 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.582181 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.582268 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.582398 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.582508 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.582592 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:54:57.582705 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.579148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41569 (local address 127.0.98.3:45489)
0620 08:54:57.579765 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.579776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:57.579863 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:57.580081 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:57.580083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:57.580097 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:54:57.580202 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.580206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.580875 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.580878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.581658 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:57.581666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.582300 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.582303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:57.582394 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.582415 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:54:57.582487 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:54:57.582568 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":266,"thread_start_us":138,"threads_started":1}
I20260620 08:54:57.582860 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389fcee462a6448286ba80071cf6c11a" candidate_uuid: "24ac612e4bde4f8eacb49dcf0444e669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75703e5624545ee8110fbe964699b9c" is_pre_election: true
I20260620 08:54:57.583141 4637 raft_consensus.cc:2454] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ac612e4bde4f8eacb49dcf0444e669 in term 0.
I20260620 08:54:57.583280 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389fcee462a6448286ba80071cf6c11a" candidate_uuid: "24ac612e4bde4f8eacb49dcf0444e669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" is_pre_election: true
I20260620 08:54:57.583527 4770 raft_consensus.cc:2454] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ac612e4bde4f8eacb49dcf0444e669 in term 0.
I20260620 08:54:57.583597 4836 leader_election.cc:302] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ac612e4bde4f8eacb49dcf0444e669, b75703e5624545ee8110fbe964699b9c; no voters:
I20260620 08:54:57.583863 4974 raft_consensus.cc:2781] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:57.583942 4974 raft_consensus.cc:482] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:57.583974 4974 raft_consensus.cc:3037] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:57.585021 4974 raft_consensus.cc:504] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.585345 4974 leader_election.cc:288] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [CANDIDATE]: Term 1 election: Requested vote from peers b75703e5624545ee8110fbe964699b9c (127.0.98.1:32973), 2ed0743c3ba0455290d35484ad0a3f3c (127.0.98.2:41569)
I20260620 08:54:57.585667 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389fcee462a6448286ba80071cf6c11a" candidate_uuid: "24ac612e4bde4f8eacb49dcf0444e669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75703e5624545ee8110fbe964699b9c"
I20260620 08:54:57.585690 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389fcee462a6448286ba80071cf6c11a" candidate_uuid: "24ac612e4bde4f8eacb49dcf0444e669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed0743c3ba0455290d35484ad0a3f3c"
I20260620 08:54:57.585784 4637 raft_consensus.cc:3037] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:57.585784 4770 raft_consensus.cc:3037] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:57.586927 4637 raft_consensus.cc:2454] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ac612e4bde4f8eacb49dcf0444e669 in term 1.
I20260620 08:54:57.586927 4770 raft_consensus.cc:2454] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ac612e4bde4f8eacb49dcf0444e669 in term 1.
I20260620 08:54:57.587205 4836 leader_election.cc:302] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ac612e4bde4f8eacb49dcf0444e669, b75703e5624545ee8110fbe964699b9c; no voters:
I20260620 08:54:57.587347 4974 raft_consensus.cc:2781] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:57.587534 4974 raft_consensus.cc:686] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 1 LEADER]: Becoming Leader. State: Replica: 24ac612e4bde4f8eacb49dcf0444e669, State: Running, Role: LEADER
I20260620 08:54:57.587837 4974 consensus_queue.cc:237] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } }
I20260620 08:54:57.590533 4493 catalog_manager.cc:5495] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ac612e4bde4f8eacb49dcf0444e669 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "24ac612e4bde4f8eacb49dcf0444e669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ac612e4bde4f8eacb49dcf0444e669" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35097 } health_report { overall_health: HEALTHY } } }
I20260620 08:54:57.618198 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.614819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.0.98.3:35097)
0620 08:54:57.614951 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.614956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.614970 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:54:57.615072 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.615074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.615128 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.615217 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:54:57.615458 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.615986 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.616817 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:57.617062 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:57.617147 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:57.617430 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:54:57.617842 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:54:57.617941 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.618010 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.618064 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:54:57.623672 4770 raft_consensus.cc:1261] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 1 FOLLOWER]: Refusing update from remote peer 24ac612e4bde4f8eacb49dcf0444e669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:57.623670 4637 raft_consensus.cc:1261] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 1 FOLLOWER]: Refusing update from remote peer 24ac612e4bde4f8eacb49dcf0444e669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:57.624346 4976 consensus_queue.cc:1035] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75703e5624545ee8110fbe964699b9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:57.624536 4974 consensus_queue.cc:1035] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed0743c3ba0455290d35484ad0a3f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:57.631815 4980 mvcc.cc:205] Tried to move back new op lower bound from 7298849577459802112 to 7298849577322221568. Current Snapshot: MvccSnapshot[applied={T|T < 7298849577459802112}]
I20260620 08:54:57.633071 4981 mvcc.cc:205] Tried to move back new op lower bound from 7298849577459802112 to 7298849577322221568. Current Snapshot: MvccSnapshot[applied={T|T < 7298849577459802112}]
I20260620 08:54:57.639066 4491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:57.639250 4491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:57.641619 4491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be1417e485024767b335eb1e9e9a0153: Sending DeleteTablet for 3 replicas of tablet 389fcee462a6448286ba80071cf6c11a
I20260620 08:54:57.642369 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 389fcee462a6448286ba80071cf6c11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52568
I20260620 08:54:57.642416 4617 tablet_service.cc:1514] Processing DeleteTablet for tablet 389fcee462a6448286ba80071cf6c11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40768
I20260620 08:54:57.642875 4989 tablet_replica.cc:330] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c: stopping tablet replica
I20260620 08:54:57.642868 4883 tablet_service.cc:1514] Processing DeleteTablet for tablet 389fcee462a6448286ba80071cf6c11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56870
I20260620 08:54:57.643193 4989 raft_consensus.cc:2229] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:57.643299 4991 tablet_replica.cc:330] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: stopping tablet replica
I20260620 08:54:57.643565 4989 raft_consensus.cc:2258] T 389fcee462a6448286ba80071cf6c11a P b75703e5624545ee8110fbe964699b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:57.643599 4990 tablet_replica.cc:330] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: stopping tablet replica
I20260620 08:54:57.643873 4990 raft_consensus.cc:2229] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:57.644232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4555
I20260620 08:54:57.644156 4990 raft_consensus.cc:2258] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:57.646903 4990 ts_tablet_manager.cc:1905] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:57.647851 4991 raft_consensus.cc:2229] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:54:57.648319 4991 raft_consensus.cc:2258] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:57.649681 4990 ts_tablet_manager.cc:1918] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:57.649830 4990 log.cc:1196] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/389fcee462a6448286ba80071cf6c11a
I20260620 08:54:57.650111 4991 ts_tablet_manager.cc:1905] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:57.650213 4990 ts_tablet_manager.cc:1939] T 389fcee462a6448286ba80071cf6c11a P 2ed0743c3ba0455290d35484ad0a3f3c: Deleting consensus metadata
W20260620 08:54:57.650996 4477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32973 (error 108)
I20260620 08:54:57.651631 4478 catalog_manager.cc:4841] TS 2ed0743c3ba0455290d35484ad0a3f3c (127.0.98.2:41569): tablet 389fcee462a6448286ba80071cf6c11a (table test-table [id=7927c22b8d404d84a3c17c2df92360ec]) successfully deleted
I20260620 08:54:57.651906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4688
I20260620 08:54:57.652609 4991 ts_tablet_manager.cc:1918] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:57.652688 4991 log.cc:1196] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/389fcee462a6448286ba80071cf6c11a
I20260620 08:54:57.652956 4991 ts_tablet_manager.cc:1939] T 389fcee462a6448286ba80071cf6c11a P 24ac612e4bde4f8eacb49dcf0444e669: Deleting consensus metadata
I20260620 08:54:57.653862 4480 catalog_manager.cc:4841] TS 24ac612e4bde4f8eacb49dcf0444e669 (127.0.98.3:35097): tablet 389fcee462a6448286ba80071cf6c11a (table test-table [id=7927c22b8d404d84a3c17c2df92360ec]) successfully deleted
I20260620 08:54:57.660015 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.659709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32973 (local address 127.0.0.1:40778)
0620 08:54:57.659832 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:57.659905 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:54:57.660137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4821
W20260620 08:54:57.660208 4477 catalog_manager.cc:4584] TS b75703e5624545ee8110fbe964699b9c (127.0.98.1:32973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 389fcee462a6448286ba80071cf6c11a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32973: connect: Connection refused (error 111)
I20260620 08:54:57.665210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4462
2026-06-20T08:54:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2144 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T08:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:57Z Disabled control of system clock
I20260620 08:54:57.693125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46267
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:54:57.797190 4996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:57.797432 4996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:57.797467 4996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:57.801363 4996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:57.801415 4996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:57.801429 4996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:57.801445 4996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:57.801458 4996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:54:57.806051 4996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:57.806958 4996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:57.808245 4996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:57.816452 5001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:57.816463 4996 server_base.cc:1034] running on GCE node
W20260620 08:54:57.816416 5002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:57.816407 5004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:57.817063 4996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:57.818060 4996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:57.819211 4996 hybrid_clock.cc:648] HybridClock initialized: now 1781945697819191 us; error 36 us; skew 500 ppm
I20260620 08:54:57.820887 4996 webserver.cc:460] Webserver started at http://127.0.98.62:33693/ using document root <none> and password file <none>
I20260620 08:54:57.821395 4996 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:57.821456 4996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:57.821635 4996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:57.823520 4996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "ab3c432a82dc4073bed412e6e3cad2c8"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "ef9e69bd7f413b2caefd9ebdfbab485f"
server_key_iv: "80db6fe09a99de185ea059045edf7521"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.824033 4996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab3c432a82dc4073bed412e6e3cad2c8"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "ef9e69bd7f413b2caefd9ebdfbab485f"
server_key_iv: "80db6fe09a99de185ea059045edf7521"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.827525 4996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:57.829946 5010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:57.831096 4996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:57.831209 4996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "ab3c432a82dc4073bed412e6e3cad2c8"
format_stamp: "Formatted at 2026-06-20 08:54:57 on dist-test-slave-fb1v"
server_key: "ef9e69bd7f413b2caefd9ebdfbab485f"
server_key_iv: "80db6fe09a99de185ea059045edf7521"
server_key_version: "encryptionkey@0"
I20260620 08:54:57.831295 4996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:57.835448 4996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:57.836035 4996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:57.836182 4996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:57.842365 4996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46267
I20260620 08:54:57.842392 5062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46267 every 8 connection(s)
I20260620 08:54:57.843186 4996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:57.846060 5063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:57.847781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 4996
I20260620 08:54:57.847894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:57.848122 392 external_mini_cluster.cc:1428] Setting key c5b44397556b110684d7b497d1816275
I20260620 08:54:57.851770 5063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8: Bootstrap starting.
I20260620 08:54:57.853248 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:57.849367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.0.98.62:46267)
0620 08:54:57.849948 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:57.849961 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:57.849992 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:54:57.850768 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:57.850785 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:57.851113 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:57.851411 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:57.852140 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:57.852145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:57.852160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:57.852170 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:57.852218 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:57.852234 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:57.852360 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:57.852424 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 08:54:57.852640 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260620 08:54:57.853912 5063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:57.854734 5063 log.cc:824] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:57.856568 5063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8: No bootstrap required, opened a new log
I20260620 08:54:57.859949 5063 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260620 08:54:57.860152 5063 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:57.860196 5063 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3c432a82dc4073bed412e6e3cad2c8, State: Initialized, Role: FOLLOWER
I20260620 08:54:57.860544 5063 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260620 08:54:57.860661 5063 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:57.860698 5063 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:57.860762 5063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:57.861513 5063 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260620 08:54:57.861835 5063 leader_election.cc:302] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3c432a82dc4073bed412e6e3cad2c8; no voters:
I20260620 08:54:57.862143 5063 leader_election.cc:288] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:57.862205 5068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:57.862393 5068 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [term 1 LEADER]: Becoming Leader. State: Replica: ab3c432a82dc4073bed412e6e3cad2c8, State: Running, Role: LEADER
I20260620 08:54:57.862676 5068 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260620 08:54:57.863343 5063 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:57.864270 5069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3c432a82dc4073bed412e6e3cad2c8. Latest consensus state: current_term: 1 leader_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } } }
I20260620 08:54:57.864318 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3c432a82dc4073bed412e6e3cad2c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } } }
I20260620 08:54:57.864470 5069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:57.864555 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:57.865254 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:57.868021 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:57.872982 5077 catalog_manager.cc:1347] Generated new cluster ID: 72644b74f4f740f4adaae514d242f4e8
I20260620 08:54:57.873054 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:57.892807 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:57.893095 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:57.903366 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8: Generated new TSK 0
I20260620 08:54:57.904053 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:54:57.912380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:54:58.020416 5087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:58.020668 5087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:58.020706 5087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:58.024691 5087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:58.024744 5087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:58.024801 5087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:54:58.029559 5087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:58.030555 5087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:58.031879 5087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:58.041342 5092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:58.041329 5095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.041424 5087 server_base.cc:1034] running on GCE node
W20260620 08:54:58.041664 5093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.042021 5087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:58.042642 5087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:58.043804 5087 hybrid_clock.cc:648] HybridClock initialized: now 1781945698043786 us; error 31 us; skew 500 ppm
I20260620 08:54:58.045590 5087 webserver.cc:460] Webserver started at http://127.0.98.1:35713/ using document root <none> and password file <none>
I20260620 08:54:58.046118 5087 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:58.046183 5087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:58.046367 5087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:58.048346 5087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a20dba58e064742921ebcd40af31ebe"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "a93a40b36899477ed64a57cd851ff319"
server_key_iv: "a10ef3eb7c59d6f8b480ec594c67482e"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.048844 5087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a20dba58e064742921ebcd40af31ebe"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "a93a40b36899477ed64a57cd851ff319"
server_key_iv: "a10ef3eb7c59d6f8b480ec594c67482e"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.052171 5087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:58.054311 5101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.055433 5087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:58.055548 5087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "8a20dba58e064742921ebcd40af31ebe"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "a93a40b36899477ed64a57cd851ff319"
server_key_iv: "a10ef3eb7c59d6f8b480ec594c67482e"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.055668 5087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:58.059868 5087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:58.060395 5087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:58.060544 5087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:58.061025 5087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:58.062095 5087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:58.062150 5087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.062186 5087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:58.062206 5087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.070107 5087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36435
I20260620 08:54:58.070158 5214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36435 every 8 connection(s)
I20260620 08:54:58.070976 5087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:54:58.076352 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.072675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46905 (local address 127.0.98.62:46267)
0620 08:54:58.072822 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.072826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.073755 (+ 929us) server_negotiation.cc:407] Connection header received
0620 08:54:58.074901 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.074903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.074938 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.075008 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.076031 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.076034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.076036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.076045 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.076062 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.076069 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.076134 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.076219 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.076250 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:54:58.077131 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.073033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46267 (local address 127.0.98.1:46905)
0620 08:54:58.073621 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.073658 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.074741 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.075131 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.075137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.075216 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.075796 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.075798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.075834 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.075897 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.076127 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.076131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.076135 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.076249 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.076424 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260620 08:54:58.077857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5087
I20260620 08:54:58.077978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:54:58.078210 5215 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46267
I20260620 08:54:58.078277 392 external_mini_cluster.cc:1428] Setting key 83106a9942b36d54fc607de7af35d933
I20260620 08:54:58.078413 5215 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:58.078810 5215 heartbeater.cc:511] Master 127.0.98.62:46267 requested a full tablet report, sending...
I20260620 08:54:58.080117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 08:54:58.080291 5027 ts_manager.cc:194] Registered new tserver with Master: 8a20dba58e064742921ebcd40af31ebe (127.0.98.1:36435)
W20260620 08:54:58.184612 5218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:58.184842 5218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:58.184886 5218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:58.188778 5218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:58.188829 5218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:58.188889 5218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:54:58.193394 5218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:58.194331 5218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:58.195714 5218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:58.204804 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:58.204787 5223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.204921 5218 server_base.cc:1034] running on GCE node
W20260620 08:54:58.204787 5226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.205467 5218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:58.206060 5218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:58.207212 5218 hybrid_clock.cc:648] HybridClock initialized: now 1781945698207193 us; error 31 us; skew 500 ppm
I20260620 08:54:58.208882 5218 webserver.cc:460] Webserver started at http://127.0.98.2:36737/ using document root <none> and password file <none>
I20260620 08:54:58.209416 5218 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:58.209478 5218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:58.209666 5218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:58.211676 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f08b461bd30481699471cdb928facd8"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "9ab93d2f5939adf8a5d428b3f1b0f771"
server_key_iv: "a289e758e996eb9f1a2bab5c380ebb83"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.212182 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f08b461bd30481699471cdb928facd8"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "9ab93d2f5939adf8a5d428b3f1b0f771"
server_key_iv: "a289e758e996eb9f1a2bab5c380ebb83"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.215638 5218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:54:58.217806 5232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.218894 5218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:54:58.218990 5218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "1f08b461bd30481699471cdb928facd8"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "9ab93d2f5939adf8a5d428b3f1b0f771"
server_key_iv: "a289e758e996eb9f1a2bab5c380ebb83"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.219084 5218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:58.223294 5218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:58.223875 5218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:58.224026 5218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:58.224503 5218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:58.225322 5218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:58.225364 5218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.225411 5218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:58.225437 5218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.233677 5218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36479
I20260620 08:54:58.233738 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36479 every 8 connection(s)
I20260620 08:54:58.234576 5218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:54:58.235849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5218
I20260620 08:54:58.235965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:54:58.236258 392 external_mini_cluster.cc:1428] Setting key b0931705731387d28ffe0299db9add5b
I20260620 08:54:58.238127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 08:54:58.243389 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.236370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58841 (local address 127.0.98.62:46267)
0620 08:54:58.236489 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.236493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.237835 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 08:54:58.239108 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.239111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.239159 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.239244 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.242940 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.242944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.242947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.242962 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.242990 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.243009 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.243112 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.243201 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.243249 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:54:58.244231 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.236838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46267 (local address 127.0.98.2:58841)
0620 08:54:58.237708 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.237746 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.238901 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.239408 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.239416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.239526 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.242661 (+ 3135us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.242662 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.242702 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.242773 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.243089 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.243093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.243097 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.243218 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.243402 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260620 08:54:58.245486 5346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46267
I20260620 08:54:58.245704 5346 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:58.246117 5346 heartbeater.cc:511] Master 127.0.98.62:46267 requested a full tablet report, sending...
I20260620 08:54:58.247051 5027 ts_manager.cc:194] Registered new tserver with Master: 1f08b461bd30481699471cdb928facd8 (127.0.98.2:36479)
W20260620 08:54:58.346666 5349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:58.346906 5349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:58.346943 5349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:58.350929 5349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:58.350988 5349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:58.351043 5349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:54:58.355839 5349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:58.356854 5349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:58.358124 5349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:58.367946 5354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:58.367852 5357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:58.367847 5355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.368073 5349 server_base.cc:1034] running on GCE node
I20260620 08:54:58.368417 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:58.368997 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:58.370157 5349 hybrid_clock.cc:648] HybridClock initialized: now 1781945698370139 us; error 28 us; skew 500 ppm
I20260620 08:54:58.371794 5349 webserver.cc:460] Webserver started at http://127.0.98.3:45093/ using document root <none> and password file <none>
I20260620 08:54:58.372308 5349 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:58.372362 5349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:58.372511 5349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:58.374423 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbd6605914894139a1194b422ef74837"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "6e33dbbef2e9c0c711941c3af2f50a85"
server_key_iv: "95021a8393cfb4b1fa75fb0b5d850fa6"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.374917 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbd6605914894139a1194b422ef74837"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "6e33dbbef2e9c0c711941c3af2f50a85"
server_key_iv: "95021a8393cfb4b1fa75fb0b5d850fa6"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.378261 5349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 08:54:58.380506 5363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.381625 5349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:58.381722 5349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "fbd6605914894139a1194b422ef74837"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "6e33dbbef2e9c0c711941c3af2f50a85"
server_key_iv: "95021a8393cfb4b1fa75fb0b5d850fa6"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.381808 5349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:58.386152 5349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:58.386680 5349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:58.386809 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:58.387286 5349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:54:58.388131 5349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:54:58.388175 5349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.388208 5349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:54:58.388222 5349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.396729 5349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36185
I20260620 08:54:58.396750 5476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36185 every 8 connection(s)
I20260620 08:54:58.397572 5349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:54:58.403422 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.399354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58311 (local address 127.0.98.62:46267)
0620 08:54:58.399501 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.399505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.400563 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 08:54:58.401799 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.401802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.401844 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.401925 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.402989 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.402994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.402996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.403008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.403028 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.403036 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.403108 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.403244 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.403293 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:54:58.403823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5349
I20260620 08:54:58.403949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:54:58.404227 392 external_mini_cluster.cc:1428] Setting key 4419f194d8c3eaed3bbe3610d8df20af
I20260620 08:54:58.404431 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.399745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46267 (local address 127.0.98.3:58311)
0620 08:54:58.400402 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.400443 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.401609 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.402083 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.402090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.402171 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.402706 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.402708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.402758 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.402840 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.403100 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.403107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.403113 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.403280 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.403512 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":121,"threads_started":1}
I20260620 08:54:58.405500 5477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46267
I20260620 08:54:58.405710 5477 heartbeater.cc:464] Registering TS with master...
I20260620 08:54:58.406140 5477 heartbeater.cc:511] Master 127.0.98.62:46267 requested a full tablet report, sending...
I20260620 08:54:58.407073 5026 ts_manager.cc:194] Registered new tserver with Master: fbd6605914894139a1194b422ef74837 (127.0.98.3:36185)
I20260620 08:54:58.417901 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:54:58.421234 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.419707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.0.98.62:46267)
0620 08:54:58.419878 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.419883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.419993 (+ 110us) server_negotiation.cc:407] Connection header received
0620 08:54:58.420128 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.420131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.420168 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.420226 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.420889 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.420893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.420895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.420908 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.420925 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.420933 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.421001 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.421072 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.421110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 08:54:58.423982 5026 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:54:58.427071 5026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:54:58.438050 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.435104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36435 (local address 127.0.0.1:32896)
0620 08:54:58.436116 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.436134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.436234 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.436692 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.436698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.436768 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.437338 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.437340 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.437358 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.437415 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.437717 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.437726 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.437731 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.437841 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.437906 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":96,"threads_started":1}
I20260620 08:54:58.438263 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.435541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36185 (local address 127.0.0.1:54644)
0620 08:54:58.436398 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.436409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.436474 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.437352 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.437355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.437369 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.437743 (+ 374us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.437744 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.437754 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.437777 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.438043 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.438046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.438049 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.438115 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.438146 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":90,"threads_started":1}
I20260620 08:54:58.438598 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.435428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.0.98.1:36435)
0620 08:54:58.435846 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.435856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.436215 (+ 359us) server_negotiation.cc:407] Connection header received
0620 08:54:58.436349 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.436356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.436563 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.436687 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.437533 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.437538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.437550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.437561 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.437603 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.437641 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.437743 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.438375 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.438446 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":119,"threads_started":1}
I20260620 08:54:58.438678 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.435711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.3:36185)
0620 08:54:58.435991 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.436001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.436482 (+ 481us) server_negotiation.cc:407] Connection header received
0620 08:54:58.436539 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.436543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.436682 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.436788 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.437869 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.437874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.437884 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.437895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.437945 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.437964 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.438048 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.438452 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.438520 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260620 08:54:58.438964 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.435025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.0.98.2:36479)
0620 08:54:58.435374 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.435386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.436212 (+ 826us) server_negotiation.cc:407] Connection header received
0620 08:54:58.436375 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.436381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.436554 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.436683 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.438385 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.438391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.438402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.438415 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.438455 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.438489 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.438573 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.438750 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.438816 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":170,"threads_started":1}
I20260620 08:54:58.439262 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.434879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36479 (local address 127.0.0.1:39626)
0620 08:54:58.436087 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.436132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.436273 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.437528 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.437530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.437544 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.438102 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.438104 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.438112 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.438134 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.438642 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.438645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.438647 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.439124 (+ 477us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.439159 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":58,"threads_started":1}
I20260620 08:54:58.441408 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 713a7a52bd3f4d439306c482621a3ea8 (DEFAULT_TABLE table=test-table [id=69232bf8e9d2484aafb9c2948921e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:58.441476 5149 tablet_service.cc:1467] Processing CreateTablet for tablet 713a7a52bd3f4d439306c482621a3ea8 (DEFAULT_TABLE table=test-table [id=69232bf8e9d2484aafb9c2948921e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:58.441824 5280 tablet_service.cc:1467] Processing CreateTablet for tablet 713a7a52bd3f4d439306c482621a3ea8 (DEFAULT_TABLE table=test-table [id=69232bf8e9d2484aafb9c2948921e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:54:58.442467 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713a7a52bd3f4d439306c482621a3ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:58.442473 5149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713a7a52bd3f4d439306c482621a3ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:58.442708 5280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713a7a52bd3f4d439306c482621a3ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:58.448455 5494 tablet_bootstrap.cc:492] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe: Bootstrap starting.
I20260620 08:54:58.449790 5495 tablet_bootstrap.cc:492] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Bootstrap starting.
I20260620 08:54:58.450839 5494 tablet_bootstrap.cc:654] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:58.451632 5495 tablet_bootstrap.cc:654] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:58.451685 5494 log.cc:824] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:58.451972 5496 tablet_bootstrap.cc:492] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Bootstrap starting.
I20260620 08:54:58.452432 5495 log.cc:824] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:58.453558 5494 tablet_bootstrap.cc:492] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe: No bootstrap required, opened a new log
I20260620 08:54:58.453744 5494 ts_tablet_manager.cc:1397] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:54:58.454010 5496 tablet_bootstrap.cc:654] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:58.454335 5495 tablet_bootstrap.cc:492] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: No bootstrap required, opened a new log
I20260620 08:54:58.454509 5495 ts_tablet_manager.cc:1397] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:54:58.454783 5496 log.cc:824] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:58.456701 5496 tablet_bootstrap.cc:492] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: No bootstrap required, opened a new log
I20260620 08:54:58.456875 5496 ts_tablet_manager.cc:1397] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:54:58.457564 5494 raft_consensus.cc:348] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.457783 5494 raft_consensus.cc:374] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:58.457840 5494 raft_consensus.cc:729] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a20dba58e064742921ebcd40af31ebe, State: Initialized, Role: FOLLOWER
I20260620 08:54:58.458240 5494 consensus_queue.cc:260] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.458602 5495 raft_consensus.cc:348] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.458849 5495 raft_consensus.cc:374] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:58.458911 5495 raft_consensus.cc:729] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd6605914894139a1194b422ef74837, State: Initialized, Role: FOLLOWER
I20260620 08:54:58.459138 5494 ts_tablet_manager.cc:1428] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:54:58.459333 5495 consensus_queue.cc:260] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.459494 5215 heartbeater.cc:503] Master 127.0.98.62:46267 was elected leader, sending a full tablet report...
I20260620 08:54:58.460023 5477 heartbeater.cc:503] Master 127.0.98.62:46267 was elected leader, sending a full tablet report...
I20260620 08:54:58.461144 5496 raft_consensus.cc:348] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.461378 5496 raft_consensus.cc:374] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:58.461424 5496 raft_consensus.cc:729] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f08b461bd30481699471cdb928facd8, State: Initialized, Role: FOLLOWER
I20260620 08:54:58.461480 5495 ts_tablet_manager.cc:1428] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260620 08:54:58.461810 5496 consensus_queue.cc:260] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.462344 5346 heartbeater.cc:503] Master 127.0.98.62:46267 was elected leader, sending a full tablet report...
I20260620 08:54:58.462661 5496 ts_tablet_manager.cc:1428] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 08:54:58.485419 5347 tablet.cc:2369] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:54:58.513674 5500 raft_consensus.cc:482] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:54:58.513881 5500 raft_consensus.cc:504] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.515007 5500 leader_election.cc:288] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f08b461bd30481699471cdb928facd8 (127.0.98.2:36479), fbd6605914894139a1194b422ef74837 (127.0.98.3:36185)
I20260620 08:54:58.516803 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.515215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36479 (local address 127.0.98.1:41321)
0620 08:54:58.515352 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.515366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.515453 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.515785 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.515788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.515804 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.516280 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.516281 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.516293 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.516318 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.516562 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.516565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.516568 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.516648 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.516696 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 08:54:58.516827 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.515341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41321 (local address 127.0.98.2:36479)
0620 08:54:58.515463 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.515467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.515482 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:54:58.515554 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.515558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.515591 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.515706 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.516432 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.516436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.516439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.516451 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.516473 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.516484 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.516569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.516657 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.516698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:54:58.517107 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.515591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38669 (local address 127.0.98.3:36185)
0620 08:54:58.515733 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.515737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.515912 (+ 175us) server_negotiation.cc:407] Connection header received
0620 08:54:58.516047 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.516049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.516086 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.516161 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.516697 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.516700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.516702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.516712 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.516730 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.516739 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.516819 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.516937 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.517001 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:54:58.517406 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.515479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36185 (local address 127.0.98.1:38669)
0620 08:54:58.515826 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.515839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.515932 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.516183 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.516185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.516198 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:54:58.516562 (+ 364us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:54:58.516564 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:54:58.516572 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:54:58.516595 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:54:58.516815 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:54:58.516818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:54:58.516821 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:54:58.517244 (+ 423us) client_negotiation.cc:240] Negotiation successful
0620 08:54:58.517279 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260620 08:54:58.517391 5300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a7a52bd3f4d439306c482621a3ea8" candidate_uuid: "8a20dba58e064742921ebcd40af31ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f08b461bd30481699471cdb928facd8" is_pre_election: true
I20260620 08:54:58.517666 5300 raft_consensus.cc:2454] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a20dba58e064742921ebcd40af31ebe in term 0.
I20260620 08:54:58.517961 5431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a7a52bd3f4d439306c482621a3ea8" candidate_uuid: "8a20dba58e064742921ebcd40af31ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd6605914894139a1194b422ef74837" is_pre_election: true
I20260620 08:54:58.518138 5105 leader_election.cc:302] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f08b461bd30481699471cdb928facd8, 8a20dba58e064742921ebcd40af31ebe; no voters:
I20260620 08:54:58.518260 5431 raft_consensus.cc:2454] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a20dba58e064742921ebcd40af31ebe in term 0.
I20260620 08:54:58.518402 5500 raft_consensus.cc:2781] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:54:58.518501 5500 raft_consensus.cc:482] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:54:58.518554 5500 raft_consensus.cc:3037] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:58.519670 5500 raft_consensus.cc:504] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.519963 5500 leader_election.cc:288] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [CANDIDATE]: Term 1 election: Requested vote from peers 1f08b461bd30481699471cdb928facd8 (127.0.98.2:36479), fbd6605914894139a1194b422ef74837 (127.0.98.3:36185)
I20260620 08:54:58.520315 5431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a7a52bd3f4d439306c482621a3ea8" candidate_uuid: "8a20dba58e064742921ebcd40af31ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd6605914894139a1194b422ef74837"
I20260620 08:54:58.520301 5300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a7a52bd3f4d439306c482621a3ea8" candidate_uuid: "8a20dba58e064742921ebcd40af31ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f08b461bd30481699471cdb928facd8"
I20260620 08:54:58.520426 5431 raft_consensus.cc:3037] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:58.520426 5300 raft_consensus.cc:3037] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:58.521589 5431 raft_consensus.cc:2454] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a20dba58e064742921ebcd40af31ebe in term 1.
I20260620 08:54:58.521589 5300 raft_consensus.cc:2454] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a20dba58e064742921ebcd40af31ebe in term 1.
I20260620 08:54:58.521878 5105 leader_election.cc:302] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a20dba58e064742921ebcd40af31ebe, fbd6605914894139a1194b422ef74837; no voters:
I20260620 08:54:58.522054 5500 raft_consensus.cc:2781] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:58.522248 5500 raft_consensus.cc:686] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [term 1 LEADER]: Becoming Leader. State: Replica: 8a20dba58e064742921ebcd40af31ebe, State: Running, Role: LEADER
I20260620 08:54:58.522557 5500 consensus_queue.cc:237] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } }
I20260620 08:54:58.525507 5027 catalog_manager.cc:5495] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a20dba58e064742921ebcd40af31ebe (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a20dba58e064742921ebcd40af31ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a20dba58e064742921ebcd40af31ebe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36435 } health_report { overall_health: HEALTHY } } }
I20260620 08:54:58.536718 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.535500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.98.1:36435)
0620 08:54:58.535629 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.535633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.535648 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:54:58.535760 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.535763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.535818 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.535880 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.536464 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.536466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.536468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.536475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.536488 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.536494 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.536548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.536592 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.536621 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:54:58.542109 5300 raft_consensus.cc:1261] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 1 FOLLOWER]: Refusing update from remote peer 8a20dba58e064742921ebcd40af31ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:58.542101 5431 raft_consensus.cc:1261] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 1 FOLLOWER]: Refusing update from remote peer 8a20dba58e064742921ebcd40af31ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:54:58.542739 5500 consensus_queue.cc:1035] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f08b461bd30481699471cdb928facd8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:58.542966 5504 consensus_queue.cc:1035] T 713a7a52bd3f4d439306c482621a3ea8 P 8a20dba58e064742921ebcd40af31ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd6605914894139a1194b422ef74837" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:54:58.548746 5507 mvcc.cc:205] Tried to move back new op lower bound from 7298849581221638144 to 7298849581150810112. Current Snapshot: MvccSnapshot[applied={T|T < 7298849581221638144}]
I20260620 08:54:58.549614 5509 mvcc.cc:205] Tried to move back new op lower bound from 7298849581221638144 to 7298849581150810112. Current Snapshot: MvccSnapshot[applied={T|T < 7298849581221638144}]
I20260620 08:54:58.557971 5027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:58.558117 5027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:54:58.560269 5027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab3c432a82dc4073bed412e6e3cad2c8: Sending DeleteTablet for 3 replicas of tablet 713a7a52bd3f4d439306c482621a3ea8
I20260620 08:54:58.561107 5280 tablet_service.cc:1514] Processing DeleteTablet for tablet 713a7a52bd3f4d439306c482621a3ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:58 UTC) from {username='slave'} at 127.0.0.1:39626
I20260620 08:54:58.561215 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet 713a7a52bd3f4d439306c482621a3ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:58 UTC) from {username='slave'} at 127.0.0.1:32896
I20260620 08:54:58.561503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5087
I20260620 08:54:58.561645 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet 713a7a52bd3f4d439306c482621a3ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:54:58 UTC) from {username='slave'} at 127.0.0.1:54644
I20260620 08:54:58.562726 5518 tablet_replica.cc:330] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: stopping tablet replica
I20260620 08:54:58.563055 5518 raft_consensus.cc:2229] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:58.563344 5518 raft_consensus.cc:2258] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:58.563704 5516 tablet_replica.cc:330] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: stopping tablet replica
I20260620 08:54:58.563970 5516 raft_consensus.cc:2229] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:54:58.564214 5516 raft_consensus.cc:2258] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:54:58.564954 5518 ts_tablet_manager.cc:1905] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:58.565464 5516 ts_tablet_manager.cc:1905] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:54:58.567122 5518 ts_tablet_manager.cc:1918] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:58.567221 5518 log.cc:1196] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/713a7a52bd3f4d439306c482621a3ea8
I20260620 08:54:58.567497 5518 ts_tablet_manager.cc:1939] T 713a7a52bd3f4d439306c482621a3ea8 P fbd6605914894139a1194b422ef74837: Deleting consensus metadata
I20260620 08:54:58.567860 5516 ts_tablet_manager.cc:1918] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:54:58.567946 5516 log.cc:1196] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/713a7a52bd3f4d439306c482621a3ea8
I20260620 08:54:58.568255 5516 ts_tablet_manager.cc:1939] T 713a7a52bd3f4d439306c482621a3ea8 P 1f08b461bd30481699471cdb928facd8: Deleting consensus metadata
I20260620 08:54:58.568599 5014 catalog_manager.cc:4841] TS fbd6605914894139a1194b422ef74837 (127.0.98.3:36185): tablet 713a7a52bd3f4d439306c482621a3ea8 (table test-table [id=69232bf8e9d2484aafb9c2948921e010]) successfully deleted
I20260620 08:54:58.569224 5014 catalog_manager.cc:4841] TS 1f08b461bd30481699471cdb928facd8 (127.0.98.2:36479): tablet 713a7a52bd3f4d439306c482621a3ea8 (table test-table [id=69232bf8e9d2484aafb9c2948921e010]) successfully deleted
W20260620 08:54:58.569356 5012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36435 (error 108)
I20260620 08:54:58.569545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5218
I20260620 08:54:58.571373 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.571119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36435 (local address 127.0.0.1:32922)
0620 08:54:58.571235 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.571301 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 08:54:58.571499 5012 catalog_manager.cc:4584] TS 8a20dba58e064742921ebcd40af31ebe (127.0.98.1:36435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 713a7a52bd3f4d439306c482621a3ea8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36435: connect: Connection refused (error 111)
I20260620 08:54:58.574676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5349
I20260620 08:54:58.579023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 4996
2026-06-20T08:54:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (912 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 08:54:58.598340 392 mini_oidc.cc:152] Starting JWKS server
I20260620 08:54:58.599905 392 webserver.cc:460] Webserver started at https://127.0.0.1:45387/ using document root <none> and password file <none>
I20260620 08:54:58.600064 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:54:58.600309 392 webserver.cc:460] Webserver started at http://127.0.0.1:38945/ using document root <none> and password file <none>
2026-06-20T08:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:58Z Disabled control of system clock
I20260620 08:54:58.608304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44945
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45387/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/testchainca.cert with env {}
W20260620 08:54:58.713616 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:58.713837 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:58.713873 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:58.717759 5525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:58.717811 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:58.717825 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:58.717840 5525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:58.717854 5525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:54:58.717867 5525 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:54:58.717885 5525 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45387/jwks.json
I20260620 08:54:58.723006 5525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45387/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:58.723904 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:58.724773 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:58.732836 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.732897 5525 server_base.cc:1034] running on GCE node
W20260620 08:54:58.732839 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:58.732816 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:58.733510 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:58.734529 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:58.735679 5525 hybrid_clock.cc:648] HybridClock initialized: now 1781945698735662 us; error 33 us; skew 500 ppm
I20260620 08:54:58.737466 5525 webserver.cc:460] Webserver started at http://127.0.98.62:38051/ using document root <none> and password file <none>
I20260620 08:54:58.737993 5525 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:58.738055 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:58.738242 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:58.740198 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "318f42b737dc469dac0c33b1b85db0f7"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "6735e69fc032dd61540ec02bc3297076"
server_key_iv: "b02572fc022ce9249533ee1c0f2d4e0b"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.740693 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "318f42b737dc469dac0c33b1b85db0f7"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "6735e69fc032dd61540ec02bc3297076"
server_key_iv: "b02572fc022ce9249533ee1c0f2d4e0b"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.744082 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:54:58.746415 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:58.747575 5525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:58.747716 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "318f42b737dc469dac0c33b1b85db0f7"
format_stamp: "Formatted at 2026-06-20 08:54:58 on dist-test-slave-fb1v"
server_key: "6735e69fc032dd61540ec02bc3297076"
server_key_iv: "b02572fc022ce9249533ee1c0f2d4e0b"
server_key_version: "encryptionkey@0"
I20260620 08:54:58.747817 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:58.762020 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:58.762641 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:58.762784 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:58.769009 5525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44945
I20260620 08:54:58.769028 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44945 every 8 connection(s)
I20260620 08:54:58.769846 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:58.772894 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:58.773193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5525
I20260620 08:54:58.773307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:58.773589 392 external_mini_cluster.cc:1428] Setting key 4d1fccb5ea18f74b7e24ea01e9035a5c
I20260620 08:54:58.779578 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7: Bootstrap starting.
I20260620 08:54:58.781720 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:58.782073 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.774753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.0.98.62:44945)
0620 08:54:58.775291 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.775306 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.775338 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:54:58.776217 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.776234 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.776604 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.776909 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:58.777306 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:58.778221 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:58.779051 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:58.779505 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:58.779678 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:58.780440 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:58.780445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:58.780456 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:58.780807 (+ 351us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:58.780862 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:58.780881 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:58.780973 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:58.781086 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 08:54:58.781320 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20260620 08:54:58.782428 5592 log.cc:824] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:58.784399 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7: No bootstrap required, opened a new log
I20260620 08:54:58.784678 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:54:58.788019 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318f42b737dc469dac0c33b1b85db0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260620 08:54:58.788225 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:58.788260 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318f42b737dc469dac0c33b1b85db0f7, State: Initialized, Role: FOLLOWER
I20260620 08:54:58.788684 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318f42b737dc469dac0c33b1b85db0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260620 08:54:58.788823 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:58.788888 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:58.788983 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:58.789795 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318f42b737dc469dac0c33b1b85db0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260620 08:54:58.790074 5592 leader_election.cc:302] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 318f42b737dc469dac0c33b1b85db0f7; no voters:
I20260620 08:54:58.790330 5592 leader_election.cc:288] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:58.790458 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:58.790808 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [term 1 LEADER]: Becoming Leader. State: Replica: 318f42b737dc469dac0c33b1b85db0f7, State: Running, Role: LEADER
I20260620 08:54:58.791114 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318f42b737dc469dac0c33b1b85db0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260620 08:54:58.791405 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:58.792588 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "318f42b737dc469dac0c33b1b85db0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318f42b737dc469dac0c33b1b85db0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260620 08:54:58.792713 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:58.793040 5606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:58.793037 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318f42b737dc469dac0c33b1b85db0f7. Latest consensus state: current_term: 1 leader_uuid: "318f42b737dc469dac0c33b1b85db0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318f42b737dc469dac0c33b1b85db0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260620 08:54:58.793972 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:58.795573 5606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:58.800836 5606 catalog_manager.cc:1347] Generated new cluster ID: f5fa1a7a01144dbba7bf8e310d6142f1
I20260620 08:54:58.800884 5606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:58.807332 5606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:58.808166 5606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:58.819008 5606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 318f42b737dc469dac0c33b1b85db0f7: Generated new TSK 0
I20260620 08:54:58.819634 5606 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 08:54:58.848054 5622 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 08:54:58.848142 5622 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 08:54:58.848786 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:58.831947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.0.98.62:44945)
0620 08:54:58.832066 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:58.832070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:58.832232 (+ 162us) server_negotiation.cc:407] Connection header received
0620 08:54:58.832411 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:58.832421 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:58.832474 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:58.832569 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:54:58.832799 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:58.833601 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:58.834472 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:58.834808 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:58.834894 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:58.835165 (+ 271us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:54:58.848401 (+ 13236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 08:54:58.848653 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 08:54:58.848843 5595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 08:54:58.849804 5621 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:54:58.831827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.0.1:50286)
0620 08:54:58.832160 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:58.832173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:58.832276 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:58.832587 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:58.832590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:58.832596 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:54:58.832713 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:58.832718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:58.833722 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:58.833725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:58.834373 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:58.834381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:58.834893 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:58.834908 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:58.834973 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:58.835012 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:54:58.848600 (+ 13588us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 08:54:58.848778 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 08:54:58.850533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5525
2026-06-20T08:54:58Z chronyd exiting
I20260620 08:54:58.886775 392 test_util.cc:175] -----------------------------------------------
I20260620 08:54:58.886852 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945671001899-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (289 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 08:54:58.887863 392 mini_oidc.cc:152] Starting JWKS server
I20260620 08:54:58.888675 392 webserver.cc:460] Webserver started at https://127.0.0.1:35945/ using document root <none> and password file <none>
I20260620 08:54:58.888768 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:54:58.888994 392 webserver.cc:460] Webserver started at http://127.0.0.1:41047/ using document root <none> and password file <none>
2026-06-20T08:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:58Z Disabled control of system clock
I20260620 08:54:58.897224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40453
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/testchainca.cert with env {}
W20260620 08:54:59.006532 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:59.006805 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:59.006857 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:59.011281 5628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:59.011415 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:59.011435 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:59.011456 5628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:59.011478 5628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:54:59.011511 5628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:54:59.011531 5628 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35945/jwks.json
I20260620 08:54:59.020236 5628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:59.021361 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:59.022408 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:59.030536 5633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:59.030536 5636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:59.030608 5628 server_base.cc:1034] running on GCE node
W20260620 08:54:59.030557 5634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:59.031212 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:59.032284 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:59.033434 5628 hybrid_clock.cc:648] HybridClock initialized: now 1781945699033417 us; error 31 us; skew 500 ppm
I20260620 08:54:59.035135 5628 webserver.cc:460] Webserver started at http://127.0.98.62:39391/ using document root <none> and password file <none>
I20260620 08:54:59.035671 5628 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:59.035733 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:59.035902 5628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:59.037775 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "ee242bd9d385466db4f31e4c3e7d6932"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "c687b4cfbb6f00eaa20f83ced24a2e75"
server_key_iv: "537282311f441ace954c321b18818e0b"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.038273 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee242bd9d385466db4f31e4c3e7d6932"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "c687b4cfbb6f00eaa20f83ced24a2e75"
server_key_iv: "537282311f441ace954c321b18818e0b"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.041574 5628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:59.043941 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:59.045085 5628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:59.045181 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "ee242bd9d385466db4f31e4c3e7d6932"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "c687b4cfbb6f00eaa20f83ced24a2e75"
server_key_iv: "537282311f441ace954c321b18818e0b"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.045275 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:59.061651 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:59.062237 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:59.062386 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:59.068603 5628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40453
I20260620 08:54:59.068620 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40453 every 8 connection(s)
I20260620 08:54:59.069444 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:59.072352 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:59.072412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5628
I20260620 08:54:59.072526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:59.072820 392 external_mini_cluster.cc:1428] Setting key ecad9ee591452ac08825a9e4f860045f
I20260620 08:54:59.078704 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932: Bootstrap starting.
I20260620 08:54:59.080986 5695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:59.081625 5695 log.cc:824] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:59.081688 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:59.074088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.0.98.62:40453)
0620 08:54:59.074716 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:59.074731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:59.074766 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:54:59.075470 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:59.075487 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:59.075923 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:59.076207 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:59.076550 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.077487 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.078635 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.079130 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.079260 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.080082 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:59.080087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:59.080099 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:59.080468 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:59.080522 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:59.080540 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:59.080615 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:59.080751 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 08:54:59.080989 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260620 08:54:59.083416 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932: No bootstrap required, opened a new log
I20260620 08:54:59.084174 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:54:59.086786 5695 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee242bd9d385466db4f31e4c3e7d6932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40453 } }
I20260620 08:54:59.086990 5695 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:59.087050 5695 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee242bd9d385466db4f31e4c3e7d6932, State: Initialized, Role: FOLLOWER
I20260620 08:54:59.087414 5695 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee242bd9d385466db4f31e4c3e7d6932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40453 } }
I20260620 08:54:59.087531 5695 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:59.087590 5695 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:59.087702 5695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:59.088466 5695 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee242bd9d385466db4f31e4c3e7d6932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40453 } }
I20260620 08:54:59.088759 5695 leader_election.cc:302] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee242bd9d385466db4f31e4c3e7d6932; no voters:
I20260620 08:54:59.089063 5695 leader_election.cc:288] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:59.089128 5700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:59.089354 5700 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [term 1 LEADER]: Becoming Leader. State: Replica: ee242bd9d385466db4f31e4c3e7d6932, State: Running, Role: LEADER
I20260620 08:54:59.089618 5700 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee242bd9d385466db4f31e4c3e7d6932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40453 } }
I20260620 08:54:59.090145 5695 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:59.090610 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee242bd9d385466db4f31e4c3e7d6932. Latest consensus state: current_term: 1 leader_uuid: "ee242bd9d385466db4f31e4c3e7d6932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee242bd9d385466db4f31e4c3e7d6932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40453 } } }
I20260620 08:54:59.090725 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:59.090946 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee242bd9d385466db4f31e4c3e7d6932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee242bd9d385466db4f31e4c3e7d6932" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40453 } } }
I20260620 08:54:59.091030 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:59.091501 5705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:59.094141 5705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:59.099475 5705 catalog_manager.cc:1347] Generated new cluster ID: 5c6495d582e4477798f9492ed4986311
I20260620 08:54:59.099529 5705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:59.115522 5705 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:59.116703 5705 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:59.132845 5705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee242bd9d385466db4f31e4c3e7d6932: Generated new TSK 0
I20260620 08:54:59.133528 5705 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 08:54:59.158113 5725 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 08:54:59.158191 5725 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 08:54:59.158875 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:59.142868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.62:40453)
0620 08:54:59.142985 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:59.142990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:59.143141 (+ 151us) server_negotiation.cc:407] Connection header received
0620 08:54:59.143275 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:59.143278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:59.143331 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:59.143455 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:54:59.143736 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.144526 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.145385 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.145730 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.145827 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.146112 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:54:59.158476 (+ 12364us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 08:54:59.158736 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 08:54:59.158938 5698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 08:54:59.159058 5724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:54:59.142766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40453 (local address 127.0.0.1:53076)
0620 08:54:59.143066 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:59.143078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:59.143172 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:59.143491 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:59.143494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:59.143502 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:54:59.143633 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:59.143639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.144650 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.144653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:59.145284 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:59.145291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.145833 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.145850 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:59.145924 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.145957 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:54:59.158694 (+ 12737us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 08:54:59.158851 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260620 08:54:59.159765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5628
2026-06-20T08:54:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 08:54:59.197666 392 mini_oidc.cc:152] Starting JWKS server
I20260620 08:54:59.198464 392 webserver.cc:460] Webserver started at https://127.0.0.1:32847/ using document root <none> and password file <none>
I20260620 08:54:59.198541 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:54:59.198743 392 webserver.cc:460] Webserver started at http://127.0.0.1:40019/ using document root <none> and password file <none>
2026-06-20T08:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:59Z Disabled control of system clock
I20260620 08:54:59.206383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34037
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32847/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 08:54:59.310595 5731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:59.310839 5731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:59.310876 5731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:59.314929 5731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:59.314993 5731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:59.315009 5731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:59.315025 5731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:59.315039 5731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:54:59.315053 5731 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:54:59.315065 5731 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32847/jwks.json
I20260620 08:54:59.319780 5731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32847/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:59.320899 5731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:59.322193 5731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:59.330029 5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:59.330020 5736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:59.330020 5737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:59.330399 5731 server_base.cc:1034] running on GCE node
I20260620 08:54:59.330984 5731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:59.332087 5731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:59.333250 5731 hybrid_clock.cc:648] HybridClock initialized: now 1781945699333228 us; error 35 us; skew 500 ppm
I20260620 08:54:59.335003 5731 webserver.cc:460] Webserver started at http://127.0.98.62:34955/ using document root <none> and password file <none>
I20260620 08:54:59.335526 5731 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:59.335590 5731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:59.335808 5731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:59.337752 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "c816fa765a304c9f820afb243031d780"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "077c3d74f22d5c4dfadec73b02777493"
server_key_iv: "c26c139def84978ceef38da1d0564aa4"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.338299 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "c816fa765a304c9f820afb243031d780"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "077c3d74f22d5c4dfadec73b02777493"
server_key_iv: "c26c139def84978ceef38da1d0564aa4"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.341782 5731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:59.344229 5745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:59.345517 5731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:59.345639 5731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "c816fa765a304c9f820afb243031d780"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "077c3d74f22d5c4dfadec73b02777493"
server_key_iv: "c26c139def84978ceef38da1d0564aa4"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.345736 5731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:59.361310 5731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:59.361986 5731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:59.362149 5731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:59.368497 5731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34037
I20260620 08:54:59.368513 5797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34037 every 8 connection(s)
I20260620 08:54:59.369369 5731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:59.371526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5731
I20260620 08:54:59.371645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:59.371923 392 external_mini_cluster.cc:1428] Setting key 2d56175ed8077667d0f4ed11285d5eb9
I20260620 08:54:59.372562 5798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:59.379184 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780: Bootstrap starting.
I20260620 08:54:59.381160 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:59.373171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.98.62:34037)
0620 08:54:59.373856 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:59.373870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:59.373909 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:54:59.374620 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:59.374636 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:59.374999 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:59.375262 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:59.375588 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.376695 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.377999 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.378463 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.378578 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.379456 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:59.379462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:59.379474 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:59.379895 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:59.379955 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:59.379976 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:59.380062 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:59.380200 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 08:54:59.380458 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":143,"threads_started":1}
I20260620 08:54:59.381546 5798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:59.382239 5798 log.cc:824] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:59.384073 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:54:59.384498 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780: No bootstrap required, opened a new log
I20260620 08:54:59.388093 5798 raft_consensus.cc:348] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c816fa765a304c9f820afb243031d780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260620 08:54:59.388329 5798 raft_consensus.cc:374] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:59.388392 5798 raft_consensus.cc:729] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c816fa765a304c9f820afb243031d780, State: Initialized, Role: FOLLOWER
I20260620 08:54:59.388808 5798 consensus_queue.cc:260] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c816fa765a304c9f820afb243031d780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260620 08:54:59.388947 5798 raft_consensus.cc:388] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:59.389015 5798 raft_consensus.cc:482] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:59.389115 5798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:59.390023 5798 raft_consensus.cc:504] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c816fa765a304c9f820afb243031d780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260620 08:54:59.390352 5798 leader_election.cc:302] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c816fa765a304c9f820afb243031d780; no voters:
I20260620 08:54:59.390654 5798 leader_election.cc:288] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:59.390779 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:59.390977 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [term 1 LEADER]: Becoming Leader. State: Replica: c816fa765a304c9f820afb243031d780, State: Running, Role: LEADER
I20260620 08:54:59.391261 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c816fa765a304c9f820afb243031d780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260620 08:54:59.391937 5798 sys_catalog.cc:564] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:59.392853 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c816fa765a304c9f820afb243031d780. Latest consensus state: current_term: 1 leader_uuid: "c816fa765a304c9f820afb243031d780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c816fa765a304c9f820afb243031d780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } }
I20260620 08:54:59.392984 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:59.393231 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c816fa765a304c9f820afb243031d780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c816fa765a304c9f820afb243031d780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } }
I20260620 08:54:59.393308 5812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:59.393314 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:59.396286 5812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:59.401631 5812 catalog_manager.cc:1347] Generated new cluster ID: 2f32f2d6a50e48a8b967d5e34b663029
I20260620 08:54:59.401701 5812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:59.408949 5812 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:59.409762 5812 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:59.415665 5812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c816fa765a304c9f820afb243031d780: Generated new TSK 0
I20260620 08:54:59.416290 5812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:54:59.428164 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:59.423344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.62:34037)
0620 08:54:59.423457 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:59.423462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:59.423592 (+ 130us) server_negotiation.cc:407] Connection header received
0620 08:54:59.423767 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:59.423770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:59.423823 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:59.423940 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:54:59.424111 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.424903 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.425718 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.426064 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.426160 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.426425 (+ 265us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:54:59.427798 (+ 1373us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32847/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 08:54:59.428025 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32847/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 08:54:59.428229 5801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32847/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 08:54:59.429973 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:54:59.423250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.0.1:42356)
0620 08:54:59.423518 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:59.423531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:59.423649 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:59.423923 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:59.423925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:59.423932 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:54:59.424030 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:59.424034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.425010 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.425012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:59.425623 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:59.425630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.426160 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.426182 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:59.426249 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.426273 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:54:59.427985 (+ 1712us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32847/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 08:54:59.429802 (+ 1817us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32847/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1590,"thread_start_us":96,"threads_started":1}
I20260620 08:54:59.430419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5731
E20260620 08:54:59.442387 5828 webserver.cc:533] Webserver: sslize failed:
E20260620 08:54:59.442456 5828 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T08:54:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (270 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T08:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:54:59Z Disabled control of system clock
I20260620 08:54:59.475345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41627
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41627
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/testchainca.cert with env {}
W20260620 08:54:59.579062 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:54:59.579349 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:54:59.579391 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:54:59.583369 5832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:54:59.583427 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:54:59.583442 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:54:59.583457 5832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:54:59.583470 5832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:54:59.583489 5832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:54:59.583516 5832 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 08:54:59.588734 5832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:54:59.589725 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:54:59.590651 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:54:59.598388 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:54:59.598387 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:59.598577 5832 server_base.cc:1034] running on GCE node
W20260620 08:54:59.598436 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:54:59.599185 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:54:59.600267 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:54:59.601454 5832 hybrid_clock.cc:648] HybridClock initialized: now 1781945699601427 us; error 40 us; skew 500 ppm
I20260620 08:54:59.603153 5832 webserver.cc:460] Webserver started at http://127.0.98.62:41313/ using document root <none> and password file <none>
I20260620 08:54:59.603715 5832 fs_manager.cc:359] Metadata directory not provided
I20260620 08:54:59.603781 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:54:59.603937 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:54:59.605856 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "ae6054f4b0394acd885366ed2fad1347"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "a0098453c712fd3a0df1f3adb47ca1f1"
server_key_iv: "018dfcdd7a91c2dfb30d19e83910b31b"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.606330 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae6054f4b0394acd885366ed2fad1347"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "a0098453c712fd3a0df1f3adb47ca1f1"
server_key_iv: "018dfcdd7a91c2dfb30d19e83910b31b"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.609807 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:54:59.612123 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:54:59.613277 5832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:54:59.613389 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "ae6054f4b0394acd885366ed2fad1347"
format_stamp: "Formatted at 2026-06-20 08:54:59 on dist-test-slave-fb1v"
server_key: "a0098453c712fd3a0df1f3adb47ca1f1"
server_key_iv: "018dfcdd7a91c2dfb30d19e83910b31b"
server_key_version: "encryptionkey@0"
I20260620 08:54:59.613497 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:54:59.624289 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:54:59.624886 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:54:59.625049 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:54:59.631862 5832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41627
I20260620 08:54:59.631892 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41627 every 8 connection(s)
I20260620 08:54:59.632735 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:54:59.635576 5899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:54:59.640231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5832
I20260620 08:54:59.640347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:54:59.640622 392 external_mini_cluster.cc:1428] Setting key 8a23ae79ed38d71027dbd9879e568bdb
I20260620 08:54:59.641175 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347: Bootstrap starting.
I20260620 08:54:59.643653 5899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347: Neither blocks nor log segments found. Creating new log.
I20260620 08:54:59.644454 5899 log.cc:824] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347: Log is configured to *not* fsync() on all Append() calls
I20260620 08:54:59.646320 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347: No bootstrap required, opened a new log
I20260620 08:54:59.649389 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:59.642005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.0.98.62:41627)
0620 08:54:59.642656 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:59.642669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:59.642704 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:54:59.643316 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:59.643332 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:59.643757 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:59.644053 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:54:59.644381 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.645340 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.646424 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.646895 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.647004 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.647774 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:54:59.647779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:54:59.647790 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:54:59.648163 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:54:59.648216 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:54:59.648235 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:54:59.648308 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:54:59.648442 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 08:54:59.648689 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
I20260620 08:54:59.650002 5899 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6054f4b0394acd885366ed2fad1347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260620 08:54:59.650203 5899 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:54:59.650421 5899 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6054f4b0394acd885366ed2fad1347, State: Initialized, Role: FOLLOWER
I20260620 08:54:59.650893 5899 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6054f4b0394acd885366ed2fad1347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260620 08:54:59.651067 5899 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:54:59.651129 5899 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:54:59.651225 5899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:54:59.652235 5899 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6054f4b0394acd885366ed2fad1347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260620 08:54:59.652457 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:54:59.652578 5899 leader_election.cc:302] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6054f4b0394acd885366ed2fad1347; no voters:
I20260620 08:54:59.652866 5899 leader_election.cc:288] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:54:59.652966 5904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:54:59.653177 5904 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [term 1 LEADER]: Becoming Leader. State: Replica: ae6054f4b0394acd885366ed2fad1347, State: Running, Role: LEADER
I20260620 08:54:59.653482 5904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6054f4b0394acd885366ed2fad1347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260620 08:54:59.654033 5899 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:54:59.655207 5906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6054f4b0394acd885366ed2fad1347. Latest consensus state: current_term: 1 leader_uuid: "ae6054f4b0394acd885366ed2fad1347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6054f4b0394acd885366ed2fad1347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } } }
I20260620 08:54:59.655331 5906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:59.655586 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6054f4b0394acd885366ed2fad1347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6054f4b0394acd885366ed2fad1347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } } }
I20260620 08:54:59.655700 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347 [sys.catalog]: This master's current role is: LEADER
I20260620 08:54:59.655814 5913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:54:59.659206 5913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:54:59.664671 5913 catalog_manager.cc:1347] Generated new cluster ID: e45b22aec7874b858eb421f8101c1a73
I20260620 08:54:59.664745 5913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:54:59.680063 5913 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:54:59.680933 5913 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:54:59.685395 5913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6054f4b0394acd885366ed2fad1347: Generated new TSK 0
I20260620 08:54:59.685976 5913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:54:59.751219 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:54:59.696600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.0.98.62:41627)
0620 08:54:59.696716 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:54:59.696721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:54:59.696875 (+ 154us) server_negotiation.cc:407] Connection header received
0620 08:54:59.697031 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:54:59.697033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:54:59.697077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:54:59.697159 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:54:59.697361 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.697884 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.699048 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.699247 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.699312 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.699597 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:54:59.750746 (+ 51149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 08:54:59.751067 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44770: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 08:54:59.751303 5902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44770: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 08:54:59.751374 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:54:59.696527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41627 (local address 127.0.0.1:44770)
0620 08:54:59.696796 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 08:54:59.696811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:54:59.696916 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:54:59.697167 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:54:59.697170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:54:59.697177 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:54:59.697280 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:59.697286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.697980 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.697983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:59.698954 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:54:59.698963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:54:59.699326 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:54:59.699345 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:54:59.699431 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:54:59.699459 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:54:59.751009 (+ 51550us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 08:54:59.751202 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41627: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260620 08:54:59.752192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5832
2026-06-20T08:54:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (301 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (114 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:00 dist-test-slave-fb1v krb5kdc[5940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:00 dist-test-slave-fb1v krb5kdc[5940](info): set up 2 sockets
Jun 20 08:55:00 dist-test-slave-fb1v krb5kdc[5940](info): commencing operation
krb5kdc: starting...
W20260620 08:55:00.939599 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:55:01.056603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:01.160969 5953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:01.161218 5953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:01.161257 5953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:01.165282 5953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:01.165351 5953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:01.165373 5953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:01.165388 5953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:01.165402 5953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:01.170703 5953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:01.171762 5953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:01.172740 5953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:01.181105 5959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:01.181113 5958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.181285 5953 server_base.cc:1034] running on GCE node
W20260620 08:55:01.181146 5961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.181929 5953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:01.183018 5953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:01.184165 5953 hybrid_clock.cc:648] HybridClock initialized: now 1781945701184150 us; error 30 us; skew 500 ppm
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.186847 5953 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:55:01.187726 5953 webserver.cc:460] Webserver started at http://127.0.98.62:33373/ using document root <none> and password file <none>
I20260620 08:55:01.188248 5953 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:01.188311 5953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:01.188503 5953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:01.190470 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "0365db72f7ea48219cacfaedbe77dbc6"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "1ce169fcafbb44dd25a87d99003c779e"
server_key_iv: "51aeb30b6cd4469ca77fbc93d989cfc7"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.190989 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "0365db72f7ea48219cacfaedbe77dbc6"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "1ce169fcafbb44dd25a87d99003c779e"
server_key_iv: "51aeb30b6cd4469ca77fbc93d989cfc7"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.194499 5953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 08:55:01.196794 5968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.197990 5953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:01.198107 5953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "0365db72f7ea48219cacfaedbe77dbc6"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "1ce169fcafbb44dd25a87d99003c779e"
server_key_iv: "51aeb30b6cd4469ca77fbc93d989cfc7"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.198196 5953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:01.217192 5953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:01.220117 5953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:01.220291 5953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:01.226853 5953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41835
I20260620 08:55:01.227171 6020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41835 every 8 connection(s)
I20260620 08:55:01.227784 5953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:01.230402 6021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:01.232314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 5953
I20260620 08:55:01.232409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:01.232671 392 external_mini_cluster.cc:1428] Setting key 36cb43d685916ef70f8257b32a165db4
I20260620 08:55:01.237021 6021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6: Bootstrap starting.
I20260620 08:55:01.239437 6021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:01.240273 6021 log.cc:824] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:01.242100 6021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6: No bootstrap required, opened a new log
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:01.245896 6021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0365db72f7ea48219cacfaedbe77dbc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260620 08:55:01.246140 6021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:01.246209 6021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0365db72f7ea48219cacfaedbe77dbc6, State: Initialized, Role: FOLLOWER
I20260620 08:55:01.246676 6021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0365db72f7ea48219cacfaedbe77dbc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260620 08:55:01.246827 6021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:01.246896 6021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:01.246989 6021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:01.248051 6021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0365db72f7ea48219cacfaedbe77dbc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260620 08:55:01.248317 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.233979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.0.98.62:41835)
0620 08:55:01.234381 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.234400 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.234441 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:55:01.235127 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.235148 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.235553 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.235934 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.236530 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.237936 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.239121 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.239844 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.239997 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.242958 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.242964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.242979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.243007 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.245325 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.245750 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.245755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.245760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.245900 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.246127 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.246131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.246132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.246621 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.246805 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.247044 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.247323 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260620 08:55:01.248400 6021 leader_election.cc:302] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0365db72f7ea48219cacfaedbe77dbc6; no voters:
I20260620 08:55:01.248678 6021 leader_election.cc:288] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:01.248781 6026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:01.249013 6026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [term 1 LEADER]: Becoming Leader. State: Replica: 0365db72f7ea48219cacfaedbe77dbc6, State: Running, Role: LEADER
I20260620 08:55:01.249326 6026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0365db72f7ea48219cacfaedbe77dbc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260620 08:55:01.249788 6021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:01.250857 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0365db72f7ea48219cacfaedbe77dbc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0365db72f7ea48219cacfaedbe77dbc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } } }
I20260620 08:55:01.250965 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:01.251066 6028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0365db72f7ea48219cacfaedbe77dbc6. Latest consensus state: current_term: 1 leader_uuid: "0365db72f7ea48219cacfaedbe77dbc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0365db72f7ea48219cacfaedbe77dbc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } } }
I20260620 08:55:01.251139 6028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:01.251437 6035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:01.254715 6035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:01.259631 6035 catalog_manager.cc:1347] Generated new cluster ID: 44ed80c5ef1944af8bc47593b81b037f
I20260620 08:55:01.259701 6035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:01.267904 6035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:01.269196 6035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:01.277483 6035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0365db72f7ea48219cacfaedbe77dbc6: Generated new TSK 0
I20260620 08:55:01.278324 6035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:55:01.310472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:01.420913 6047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:01.421234 6047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:01.421298 6047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:01.425529 6047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:01.425594 6047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:01.425675 6047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:01.431329 6047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:01.432750 6047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:01.433831 6047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:01.444371 6052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.444489 6047 server_base.cc:1034] running on GCE node
W20260620 08:55:01.444507 6053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:01.444371 6055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.445035 6047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:01.445612 6047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:01.446770 6047 hybrid_clock.cc:648] HybridClock initialized: now 1781945701446745 us; error 45 us; skew 500 ppm
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.449386 6047 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:55:01.450312 6047 webserver.cc:460] Webserver started at http://127.0.98.1:40937/ using document root <none> and password file <none>
I20260620 08:55:01.450917 6047 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:01.450985 6047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:01.451169 6047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:01.453181 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c59a01a0ea540b1a87e6615d93bb516"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "afd3082f89ac6cdf589b68ff4b5979bf"
server_key_iv: "2af7d68b3ac10afc91de32c07cb4643d"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.453682 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c59a01a0ea540b1a87e6615d93bb516"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "afd3082f89ac6cdf589b68ff4b5979bf"
server_key_iv: "2af7d68b3ac10afc91de32c07cb4643d"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.457265 6047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 08:55:01.459733 6062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.460902 6047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:55:01.460996 6047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "0c59a01a0ea540b1a87e6615d93bb516"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "afd3082f89ac6cdf589b68ff4b5979bf"
server_key_iv: "2af7d68b3ac10afc91de32c07cb4643d"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.461097 6047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:01.477167 6047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:01.480736 6047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:01.480964 6047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:01.481745 6047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:01.482826 6047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:01.482884 6047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.482949 6047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:01.482975 6047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.492688 6175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44909 every 8 connection(s)
I20260620 08:55:01.492730 6047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44909
I20260620 08:55:01.493795 6047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:01.496760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6047
I20260620 08:55:01.496879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:01.497188 392 external_mini_cluster.cc:1428] Setting key 85f92205a38646f572b142d561735395
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:55:01.508862 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.495672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57267 (local address 127.0.98.62:41835)
0620 08:55:01.495789 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.495793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.497181 (+ 1388us) server_negotiation.cc:407] Connection header received
0620 08:55:01.498751 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.498754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.498804 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.498895 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.500231 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.500838 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.502043 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.502278 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.502367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.505074 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.505077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.505079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.505102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.506494 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.507024 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.507027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.507028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.507114 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.507745 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.507749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.507750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.507942 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.508024 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.508563 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.508699 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:01.509663 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.496310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41835 (local address 127.0.98.1:57267)
0620 08:55:01.497023 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:01.497078 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:01.498560 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:01.499044 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:01.499051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:01.499403 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:01.500061 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:01.500079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.500959 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.500963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:01.501908 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:01.501917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.502418 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.502422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:01.502626 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.503192 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:01.503207 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:01.504860 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:01.506600 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:01.506605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:01.506620 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:01.506874 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:01.507192 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:01.507194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:01.507196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:01.507325 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:01.508023 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:01.508028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:01.508300 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 08:55:01.508502 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 08:55:01.508788 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":240,"threads_started":1}
I20260620 08:55:01.510810 6176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41835
I20260620 08:55:01.511060 6176 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:01.511538 6176 heartbeater.cc:511] Master 127.0.98.62:41835 requested a full tablet report, sending...
I20260620 08:55:01.513360 5985 ts_manager.cc:194] Registered new tserver with Master: 0c59a01a0ea540b1a87e6615d93bb516 (127.0.98.1:44909)
I20260620 08:55:01.514919 5985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:55:01.526302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:01.636962 6181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:01.637223 6181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:01.637283 6181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:01.641326 6181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:01.641382 6181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:01.641460 6181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:01.646839 6181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:01.648134 6181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:01.649247 6181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:01.658735 6189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:01.658733 6186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.658883 6181 server_base.cc:1034] running on GCE node
W20260620 08:55:01.658797 6187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.659373 6181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:01.660059 6181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:01.661239 6181 hybrid_clock.cc:648] HybridClock initialized: now 1781945701661218 us; error 31 us; skew 500 ppm
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.664198 6181 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:55:01.665143 6181 webserver.cc:460] Webserver started at http://127.0.98.2:33949/ using document root <none> and password file <none>
I20260620 08:55:01.665695 6181 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:01.665761 6181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:01.665944 6181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:01.667994 6181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "4dd739ee7700412a9b4504e33f0ef623"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "45ea506d7eb14afc9f661fab7676a1a2"
server_key_iv: "4887f5a9159546f03eb0c6193666cce9"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.668514 6181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4dd739ee7700412a9b4504e33f0ef623"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "45ea506d7eb14afc9f661fab7676a1a2"
server_key_iv: "4887f5a9159546f03eb0c6193666cce9"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.672185 6181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:01.674888 6196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.676622 6181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:01.676826 6181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "4dd739ee7700412a9b4504e33f0ef623"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "45ea506d7eb14afc9f661fab7676a1a2"
server_key_iv: "4887f5a9159546f03eb0c6193666cce9"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.676985 6181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:01.686661 6181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:01.689946 6181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:01.690140 6181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:01.690757 6181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:01.691597 6181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:01.691700 6181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.691754 6181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:01.691781 6181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.700826 6181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36347
I20260620 08:55:01.700855 6309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36347 every 8 connection(s)
I20260620 08:55:01.701808 6181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:01.702415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6181
I20260620 08:55:01.702521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:55:01.702803 392 external_mini_cluster.cc:1428] Setting key 6fc07a47549b60d6b54c35815c5c8b88
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:01.722762 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.703638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58561 (local address 127.0.98.62:41835)
0620 08:55:01.703741 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.703745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.705389 (+ 1644us) server_negotiation.cc:407] Connection header received
0620 08:55:01.707559 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.707563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.707660 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.707743 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.709352 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.709890 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.711594 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.711847 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.711936 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.717748 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.717752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.717754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.717781 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.719513 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.720515 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.720518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.720520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.720614 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.721114 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.721117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.721119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.721327 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.721412 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.722465 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.722631 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:55:01.723989 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.704425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41835 (local address 127.0.98.2:58561)
0620 08:55:01.705063 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:01.705229 (+ 166us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:01.707255 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:01.707961 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:01.707970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:01.708456 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:01.709155 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:01.709179 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.710034 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.710038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:01.711453 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:01.711464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.712011 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.712016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:01.712214 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.712805 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:01.712828 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:01.717553 (+ 4725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:01.719824 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:01.719834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:01.719855 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:01.720327 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:01.720707 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:01.720710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:01.720712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:01.720931 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:01.721449 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:01.721458 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:01.722095 (+ 637us) client_negotiation.cc:769] Sending connection context
0620 08:55:01.722355 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 08:55:01.722798 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":112,"threads_started":1}
I20260620 08:55:01.725363 6310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41835
I20260620 08:55:01.725600 6310 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:01.726123 6310 heartbeater.cc:511] Master 127.0.98.62:41835 requested a full tablet report, sending...
I20260620 08:55:01.727185 5985 ts_manager.cc:194] Registered new tserver with Master: 4dd739ee7700412a9b4504e33f0ef623 (127.0.98.2:36347)
I20260620 08:55:01.727851 5985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:55:01.734979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:01.843592 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:01.843899 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:01.843962 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:01.848008 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:01.848066 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:01.848148 6315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:01.853513 6315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:01.854477 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:01.855546 6315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:01.864447 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:01.864538 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.864666 6315 server_base.cc:1034] running on GCE node
W20260620 08:55:01.864688 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:01.865090 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:01.865741 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:01.866904 6315 hybrid_clock.cc:648] HybridClock initialized: now 1781945701866884 us; error 30 us; skew 500 ppm
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.869506 6315 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:55:01.870282 6315 webserver.cc:460] Webserver started at http://127.0.98.3:38841/ using document root <none> and password file <none>
I20260620 08:55:01.870797 6315 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:01.870851 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:01.871007 6315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:01.872959 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "2fbd7dccf01343a5a0277de700eedea9"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "b021c65e11422cef3f2c3bcd2dba8c6e"
server_key_iv: "7dd788d02884f912a7a8bfafc268abee"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.873426 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2fbd7dccf01343a5a0277de700eedea9"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "b021c65e11422cef3f2c3bcd2dba8c6e"
server_key_iv: "7dd788d02884f912a7a8bfafc268abee"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.876861 6315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:55:01.879160 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.880453 6315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:01.880568 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "2fbd7dccf01343a5a0277de700eedea9"
format_stamp: "Formatted at 2026-06-20 08:55:01 on dist-test-slave-fb1v"
server_key: "b021c65e11422cef3f2c3bcd2dba8c6e"
server_key_iv: "7dd788d02884f912a7a8bfafc268abee"
server_key_version: "encryptionkey@0"
I20260620 08:55:01.880661 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:01.894794 6315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:01.897655 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:01.897821 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:01.898381 6315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:01.899178 6315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:01.899226 6315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.899278 6315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:01.899307 6315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:01.907763 6315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41763
I20260620 08:55:01.907806 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41763 every 8 connection(s)
I20260620 08:55:01.908624 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:01.911242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6315
I20260620 08:55:01.911342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:01.911641 392 external_mini_cluster.cc:1428] Setting key 9a0bec743b6806c5150611e70790a644
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:01.922338 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.910391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38929 (local address 127.0.98.62:41835)
0620 08:55:01.910558 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.910564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.911529 (+ 965us) server_negotiation.cc:407] Connection header received
0620 08:55:01.913082 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.913086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.913138 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.913221 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.914323 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.914849 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.916176 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.916409 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.916491 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.918843 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.918846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.918848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.918871 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.920204 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.920740 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.920743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.920745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.920838 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.921221 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.921224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.921226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.921428 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.921524 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.922035 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.922141 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:55:01.923069 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.910790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41835 (local address 127.0.98.3:38929)
0620 08:55:01.911348 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:01.911404 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:01.912877 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:01.913355 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:01.913365 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:01.913742 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:01.914195 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:01.914209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.914986 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.914989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:01.916050 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:01.916061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.916573 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.916577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:01.916719 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.917296 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:01.917312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:01.918678 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:01.920314 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:01.920319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:01.920331 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:01.920592 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:01.920961 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:01.920964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:01.920966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:01.921105 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:01.921549 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:01.921555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:01.921786 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 08:55:01.921949 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 08:55:01.922206 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":98,"threads_started":1}
I20260620 08:55:01.924417 6444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41835
I20260620 08:55:01.924677 6444 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:01.925254 6444 heartbeater.cc:511] Master 127.0.98.62:41835 requested a full tablet report, sending...
I20260620 08:55:01.926333 5985 ts_manager.cc:194] Registered new tserver with Master: 2fbd7dccf01343a5a0277de700eedea9 (127.0.98.3:41763)
I20260620 08:55:01.926888 5985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38929
I20260620 08:55:01.937166 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/test-admin.keytab.
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.951298 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 08:55:01.953943 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:01.952612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.0.98.1:44909)
0620 08:55:01.952920 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.952929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.952951 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:55:01.953014 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.953019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.953196 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.953308 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.953759 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42268: BlockingRecv error: recv got EOF from 127.0.0.1:42268 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":119,"threads_started":1}
W20260620 08:55:01.953991 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.954881 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.955801 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:55:01.964947 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.956554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.0.98.1:44909)
0620 08:55:01.956687 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.956690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.956734 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:55:01.956822 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.956824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.956866 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.956932 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.957489 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.958276 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.959139 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.959522 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.959635 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.961765 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.961773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.961786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.961809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.963214 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.963637 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.963643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.963645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.963722 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.964007 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.964010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.964011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.964253 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.964431 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.964636 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.964813 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 08:55:01.968520 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:01.967581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.0.98.1:44909)
0620 08:55:01.967715 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.967719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.967781 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:55:01.967909 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.967912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.967946 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.968004 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.968424 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42294: BlockingRecv error: recv got EOF from 127.0.0.1:42294 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 08:55:01.968652 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.969718 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.970647 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:55:01.978849 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.971537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.0.98.1:44909)
0620 08:55:01.971695 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.971699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.971713 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:55:01.971798 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.971801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.971842 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.971919 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.972342 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.972865 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.973628 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.973858 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.973975 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.976035 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.976038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.976040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.976060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.977287 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.977786 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.977789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.977790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.977870 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.978114 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.978116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.978117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.978269 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.978366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.978598 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.978689 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 08:55:01.982000 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:01.980820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.0.98.1:44909)
0620 08:55:01.980928 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.980931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.981127 (+ 196us) server_negotiation.cc:407] Connection header received
0620 08:55:01.981203 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.981205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.981241 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.981305 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.981901 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42318: BlockingRecv error: recv got EOF from 127.0.0.1:42318 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 08:55:01.982247 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945671001899-392-0/krb5kdc/krb5cc)
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.983201 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:01.984364 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:55:01 dist-test-slave-fb1v krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:55:01.992722 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:01.985164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.0.98.1:44909)
0620 08:55:01.985281 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:01.985285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:01.985328 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:55:01.985429 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:01.985431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:01.985469 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:01.985540 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:01.985940 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.986423 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.987214 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:01.987419 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:01.987498 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:01.989606 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:01.989609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:01.989611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:01.989632 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:01.990878 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.991327 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.991330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.991331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.991419 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:01.991902 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:01.991905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:01.991906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:01.992068 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:01.992167 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:01.992429 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 08:55:01.992554 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:55:01.993764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6047
I20260620 08:55:01.999225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6181
I20260620 08:55:02.003811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6315
I20260620 08:55:02.008540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 5953
2026-06-20T08:55:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2027 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:02 dist-test-slave-fb1v krb5kdc[6464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:02 dist-test-slave-fb1v krb5kdc[6464](info): set up 2 sockets
Jun 20 08:55:02 dist-test-slave-fb1v krb5kdc[6464](info): commencing operation
krb5kdc: starting...
W20260620 08:55:02.979745 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 08:55:03.098539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36477
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:03.205904 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:03.206162 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:03.206218 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:03.210224 6477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:03.210278 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:03.210305 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:03.210327 6477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:03.210355 6477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:03.215505 6477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:03.216504 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:03.217443 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:03.225925 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.225925 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.225921 6485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:03.226015 6477 server_base.cc:1034] running on GCE node
I20260620 08:55:03.226677 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:03.227751 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:03.228917 6477 hybrid_clock.cc:648] HybridClock initialized: now 1781945703228905 us; error 49 us; skew 500 ppm
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:03.231539 6477 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 08:55:03.232388 6477 webserver.cc:460] Webserver started at http://127.0.98.62:40527/ using document root <none> and password file <none>
I20260620 08:55:03.232887 6477 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:03.232942 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:03.233143 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:03.235114 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "13c585760d62d16c647173849bd92b5c"
server_key_iv: "91dfe511d03dc5c9dc75392427a55cc2"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.235652 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "13c585760d62d16c647173849bd92b5c"
server_key_iv: "91dfe511d03dc5c9dc75392427a55cc2"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.239158 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:55:03.241459 6492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.242630 6477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:03.242738 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "13c585760d62d16c647173849bd92b5c"
server_key_iv: "91dfe511d03dc5c9dc75392427a55cc2"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.242825 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:03.261104 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:03.264060 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:03.264225 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:03.270633 6477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36477
I20260620 08:55:03.270889 6544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36477 every 8 connection(s)
I20260620 08:55:03.271517 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:03.274170 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:03.274394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6477
I20260620 08:55:03.274518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:03.274781 392 external_mini_cluster.cc:1428] Setting key 39efaf5c2748fb464e5b59aeb1f30176
I20260620 08:55:03.281152 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Bootstrap starting.
I20260620 08:55:03.283268 6545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Neither blocks nor log segments found. Creating new log.
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:03.284018 6545 log.cc:824] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:03.285864 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: No bootstrap required, opened a new log
I20260620 08:55:03.288343 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:03.276159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.0.98.62:36477)
0620 08:55:03.276576 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:03.276596 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:03.276645 (+ 49us) server_negotiation.cc:407] Connection header received
0620 08:55:03.277372 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:03.277395 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:03.277795 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:03.278160 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:03.278800 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.280390 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.281287 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.281745 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.281908 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:03.284245 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:03.284250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:03.284262 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:03.284288 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:03.285766 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:03.286189 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:03.286194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:03.286198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:03.286300 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:03.286545 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:03.286549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:03.286551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:03.286937 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:03.287107 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:03.287373 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 08:55:03.287667 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260620 08:55:03.289853 6545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } }
I20260620 08:55:03.290262 6545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:03.290346 6545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3aed0df60b4a10ae10f6ab759f4ce7, State: Initialized, Role: FOLLOWER
I20260620 08:55:03.290774 6545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } }
I20260620 08:55:03.290904 6545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:03.290961 6545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:03.291031 6545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:03.291926 6545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } }
I20260620 08:55:03.292228 6545 leader_election.cc:302] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3aed0df60b4a10ae10f6ab759f4ce7; no voters:
I20260620 08:55:03.292479 6545 leader_election.cc:288] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:03.292603 6550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:03.292809 6550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3aed0df60b4a10ae10f6ab759f4ce7, State: Running, Role: LEADER
I20260620 08:55:03.293138 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } }
I20260620 08:55:03.293682 6545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:03.294826 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c3aed0df60b4a10ae10f6ab759f4ce7. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } }
I20260620 08:55:03.294966 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:03.294929 6551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } }
I20260620 08:55:03.295149 6551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:03.295441 6559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:03.298934 6559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:03.304770 6559 catalog_manager.cc:1347] Generated new cluster ID: a9e19b1412e840e9b9a52a6d69159b9a
I20260620 08:55:03.304847 6559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:03.332609 6559 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:03.333570 6559 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:03.343628 6559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Generated new TSK 0
I20260620 08:55:03.344281 6559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 08:55:03.380090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36477
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:03.506639 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:03.506884 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:03.506925 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:03.510893 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:03.510952 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:03.511014 6571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:03.516199 6571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:03.517138 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:03.518059 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:03.527122 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.527180 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.527221 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:03.527273 6571 server_base.cc:1034] running on GCE node
I20260620 08:55:03.527864 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:03.528455 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:03.529606 6571 hybrid_clock.cc:648] HybridClock initialized: now 1781945703529592 us; error 29 us; skew 500 ppm
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:03.532122 6571 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 08:55:03.532887 6571 webserver.cc:460] Webserver started at http://127.0.98.1:43093/ using document root <none> and password file <none>
I20260620 08:55:03.533380 6571 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:03.533440 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:03.533618 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:03.535465 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "dbb7ef58462741c6bde02a0a9c023fa9"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "66e2ee2f56118ca77f41f383b7bd6882"
server_key_iv: "e6d91270839ce013c19bc9e90f5da3bb"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.536024 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dbb7ef58462741c6bde02a0a9c023fa9"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "66e2ee2f56118ca77f41f383b7bd6882"
server_key_iv: "e6d91270839ce013c19bc9e90f5da3bb"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.539340 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:03.541522 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.542662 6571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:03.542757 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "dbb7ef58462741c6bde02a0a9c023fa9"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "66e2ee2f56118ca77f41f383b7bd6882"
server_key_iv: "e6d91270839ce013c19bc9e90f5da3bb"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.542838 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:03.563449 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:03.567183 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:03.567471 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:03.568368 6571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:03.569425 6571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:03.569489 6571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.569550 6571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:03.569573 6571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.579058 6571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46045
I20260620 08:55:03.579078 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46045 every 8 connection(s)
I20260620 08:55:03.580034 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:03.586648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6571
I20260620 08:55:03.586786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:03.587085 392 external_mini_cluster.cc:1428] Setting key 4cc8c4057c3ba68d556bd9a99d9742a8
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:03.595312 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:03.582131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35439 (local address 127.0.98.62:36477)
0620 08:55:03.582246 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:03.582250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:03.583451 (+ 1201us) server_negotiation.cc:407] Connection header received
0620 08:55:03.584832 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:03.584835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:03.584892 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:03.584972 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:03.586168 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.586718 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.588438 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.588725 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.588836 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:03.591292 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:03.591295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:03.591297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:03.591322 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:03.592875 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:03.593555 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:03.593558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:03.593560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:03.593637 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:03.594032 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:03.594036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:03.594038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:03.594285 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:03.594381 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:03.595032 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 08:55:03.595156 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:55:03.596467 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:03.582673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.1:35439)
0620 08:55:03.583266 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:03.583312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:03.584601 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:03.585157 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:03.585165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:03.585530 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:03.586009 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:03.586032 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.587011 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.587016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:03.588275 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:03.588291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.588956 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.588962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:03.589120 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:03.589683 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:03.589700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:03.591119 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:03.593006 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:03.593013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:03.593035 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:03.593431 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:03.593730 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:03.593733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:03.593735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:03.593914 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:03.594397 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:03.594404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:03.594737 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 08:55:03.594985 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 08:55:03.595376 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":160,"threads_started":1}
I20260620 08:55:03.597612 6700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36477
I20260620 08:55:03.597833 6700 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:03.598304 6700 heartbeater.cc:511] Master 127.0.98.62:36477 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 08:55:03.599828 6509 ts_manager.cc:194] Registered new tserver with Master: dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045)
I20260620 08:55:03.600920 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 08:55:03.614341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36477
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:03.720682 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:03.720919 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:03.720960 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:03.724951 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:03.725004 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:03.725075 6705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:03.730166 6705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:03.731137 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:03.732163 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:03.741197 6713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.741195 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.741214 6710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:03.741376 6705 server_base.cc:1034] running on GCE node
I20260620 08:55:03.741828 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:03.742421 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:03.743575 6705 hybrid_clock.cc:648] HybridClock initialized: now 1781945703743565 us; error 36 us; skew 500 ppm
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:03.746100 6705 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 08:55:03.746922 6705 webserver.cc:460] Webserver started at http://127.0.98.2:38479/ using document root <none> and password file <none>
I20260620 08:55:03.747411 6705 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:03.747471 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:03.747704 6705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:03.749532 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8b39082886f433c9b1a786b1c8e4328"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "9f7c1806178bb1313c9e5adcae39bce2"
server_key_iv: "489df795bb3c43f9beb05cae556f70ba"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.750036 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8b39082886f433c9b1a786b1c8e4328"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "9f7c1806178bb1313c9e5adcae39bce2"
server_key_iv: "489df795bb3c43f9beb05cae556f70ba"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.753422 6705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:55:03.755728 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.756848 6705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:03.756950 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "d8b39082886f433c9b1a786b1c8e4328"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "9f7c1806178bb1313c9e5adcae39bce2"
server_key_iv: "489df795bb3c43f9beb05cae556f70ba"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.757040 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:03.777796 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:03.780655 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:03.780818 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:03.781359 6705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:03.782135 6705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:03.782182 6705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.782230 6705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:03.782258 6705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.790294 6705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39981
I20260620 08:55:03.790324 6833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39981 every 8 connection(s)
I20260620 08:55:03.791116 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:03.799997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6705
I20260620 08:55:03.800091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:03.800333 392 external_mini_cluster.cc:1428] Setting key b556322c3da19b1b16b470f6841396c8
I20260620 08:55:03.804219 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:03.792825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49749 (local address 127.0.98.62:36477)
0620 08:55:03.792944 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:03.792948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:03.793754 (+ 806us) server_negotiation.cc:407] Connection header received
0620 08:55:03.795144 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:03.795148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:03.795202 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:03.795274 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:03.796475 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.796980 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.798043 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.798262 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.798347 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:03.800847 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:03.800851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:03.800853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:03.800880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:03.802359 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:03.802803 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:03.802806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:03.802808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:03.802898 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:03.803179 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:03.803183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:03.803185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:03.803354 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:03.803439 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:03.803930 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 08:55:03.804065 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:03.805023 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:03.793176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:49749)
0620 08:55:03.793615 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:03.793667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:03.794953 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:03.795442 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:03.795448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:03.795881 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:03.796339 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:03.796352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.797090 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.797094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:03.797933 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:03.797942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:03.798388 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:03.798393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:03.798540 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:03.799260 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:03.799275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:03.800716 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:03.802459 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:03.802463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:03.802480 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:03.802722 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:03.802974 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:03.802976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:03.802977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:03.803091 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:03.803447 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:03.803453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:03.803715 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 08:55:03.803873 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 08:55:03.804167 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260620 08:55:03.806146 6834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36477
I20260620 08:55:03.806391 6834 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:03.806933 6834 heartbeater.cc:511] Master 127.0.98.62:36477 requested a full tablet report, sending...
I20260620 08:55:03.807920 6509 ts_manager.cc:194] Registered new tserver with Master: d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981)
I20260620 08:55:03.808449 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49749
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 08:55:03.826393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36477
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:03.933413 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:03.933640 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:03.933687 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:03.937670 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:03.937718 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:03.937777 6839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:03.942839 6839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:03.943785 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:03.944705 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:03.953723 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.953723 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:03.953799 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:03.954012 6839 server_base.cc:1034] running on GCE node
I20260620 08:55:03.954355 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:03.954916 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:03.956086 6839 hybrid_clock.cc:648] HybridClock initialized: now 1781945703956071 us; error 28 us; skew 500 ppm
Jun 20 08:55:03 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:03.958559 6839 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 08:55:03.959372 6839 webserver.cc:460] Webserver started at http://127.0.98.3:39373/ using document root <none> and password file <none>
I20260620 08:55:03.959923 6839 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:03.959988 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:03.960160 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:03.962142 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "eee75217f62c468dba0fcc893ec81db7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "8c501c9a576d9366a0afa46730282a0f"
server_key_iv: "a718fdf5aedb2e90a0d1f98aa7e255ec"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.962666 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eee75217f62c468dba0fcc893ec81db7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "8c501c9a576d9366a0afa46730282a0f"
server_key_iv: "a718fdf5aedb2e90a0d1f98aa7e255ec"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.966234 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 08:55:03.968506 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.969841 6839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:03.969967 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "eee75217f62c468dba0fcc893ec81db7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "8c501c9a576d9366a0afa46730282a0f"
server_key_iv: "a718fdf5aedb2e90a0d1f98aa7e255ec"
server_key_version: "encryptionkey@0"
I20260620 08:55:03.970058 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:03.984869 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:03.987841 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:03.987991 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:03.988512 6839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:03.989285 6839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:03.989332 6839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.989374 6839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:03.989390 6839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:03.997802 6839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45053
I20260620 08:55:03.998039 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45053 every 8 connection(s)
I20260620 08:55:03.998678 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:04.002214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6839
I20260620 08:55:04.002336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:04.002609 392 external_mini_cluster.cc:1428] Setting key a67a36b07d47b94c8a858e4d1a020025
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:04.013195 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.000438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44389 (local address 127.0.98.62:36477)
0620 08:55:04.000592 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.000597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.001395 (+ 798us) server_negotiation.cc:407] Connection header received
0620 08:55:04.002737 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.002740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.002797 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.002873 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.004664 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.005263 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.006482 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.006698 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.006778 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.009271 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.009274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.009275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.009301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.010619 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.011426 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.011431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.011432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.011560 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.012071 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.012075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.012076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.012286 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.012384 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.012968 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.013073 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:55:04.014040 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.000805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:44389)
0620 08:55:04.001247 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.001290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.002511 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.003063 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.003072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.003518 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.004477 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.004500 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.005413 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.005417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.006353 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.006365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.006847 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.006851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.007012 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.007555 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.007572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.009066 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.010804 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.010812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.010831 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.011238 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.011717 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.011721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.011723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.011921 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.012436 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.012442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.012716 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.012893 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.013193 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260620 08:55:04.015276 6968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36477
I20260620 08:55:04.015683 6968 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:04.016453 6968 heartbeater.cc:511] Master 127.0.98.62:36477 requested a full tablet report, sending...
I20260620 08:55:04.017508 6509 ts_manager.cc:194] Registered new tserver with Master: eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053)
I20260620 08:55:04.018074 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44389
I20260620 08:55:04.029126 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 08:55:04.055048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38155
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36477,127.0.98.61:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:04.159404 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:04.159684 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:04.159734 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:04.163823 6973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:04.163883 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:04.163901 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:04.163916 6973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:04.163934 6973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:04.169091 6973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36477,127.0.98.61:38155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
W20260620 08:55:04.169270 6973 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36477,127.0.98.61:38155'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 08:55:04.170096 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:04.171031 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:04.178975 6979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.179049 6973 server_base.cc:1034] running on GCE node
W20260620 08:55:04.179054 6981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.179059 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.179766 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:04.180789 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:04.181939 6973 hybrid_clock.cc:648] HybridClock initialized: now 1781945704181922 us; error 31 us; skew 500 ppm
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:04.184718 6973 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 08:55:04.185525 6973 webserver.cc:460] Webserver started at http://127.0.98.61:44151/ using document root <none> and password file <none>
I20260620 08:55:04.186031 6973 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.186092 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.186264 6973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:04.188309 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/instance:
uuid: "7f27f25fbfe44a4d949940405fa61719"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "427a64ade18606c2d15df8e3cb4951c8"
server_key_iv: "81373f9991bb258ca18eed61c21a20aa"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.188851 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal/instance:
uuid: "7f27f25fbfe44a4d949940405fa61719"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "427a64ade18606c2d15df8e3cb4951c8"
server_key_iv: "81373f9991bb258ca18eed61c21a20aa"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.192421 6973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:55:04.194865 6988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.196153 6973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:04.196275 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
uuid: "7f27f25fbfe44a4d949940405fa61719"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "427a64ade18606c2d15df8e3cb4951c8"
server_key_iv: "81373f9991bb258ca18eed61c21a20aa"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.196364 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.205852 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:04.208477 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:04.208626 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:04.214656 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38155
I20260620 08:55:04.214898 7040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38155 every 8 connection(s)
I20260620 08:55:04.215499 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
I20260620 08:55:04.218045 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:04.220410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 6973
I20260620 08:55:04.221755 7041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 08:55:04.233052 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.223599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.0.98.62:36477)
0620 08:55:04.223736 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.223740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.224081 (+ 341us) server_negotiation.cc:407] Connection header received
0620 08:55:04.224291 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.224295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.224342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.224393 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.225272 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.225771 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.226720 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.226926 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.227009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.229363 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.229366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.229369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.229395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.231002 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.231550 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.231553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.231554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.231656 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.231982 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.231985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.231987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.232240 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.232334 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.232771 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.232893 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:55:04.233860 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.221734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.0.98.61:38155)
0620 08:55:04.222136 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.222155 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.222195 (+ 40us) server_negotiation.cc:407] Connection header received
0620 08:55:04.223004 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.223024 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.223357 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.223671 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.224681 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.225753 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.226574 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.226946 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.227086 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.229827 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.229834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.229846 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.229870 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.231423 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.231847 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.231851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.231852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.231923 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.232176 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.232179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.232181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.232546 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.232673 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.232941 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.233066 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260620 08:55:04.233857 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.223543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34150)
0620 08:55:04.223969 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.224014 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.224161 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.224557 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.224562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.224985 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.225149 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.225159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.225883 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.225885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.226613 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.226621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.227048 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.227052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.227141 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.227663 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.227682 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.229230 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.231117 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.231123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.231131 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.231442 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.231740 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.231743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.231745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.231899 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.232337 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.232342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.232557 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.232714 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.233008 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":4,"spinlock_wait_cycles":2944,"thread_start_us":126,"threads_started":1}
I20260620 08:55:04.235643 7041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 08:55:04.242189 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.236077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35360)
0620 08:55:04.236176 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.236189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.236252 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.236418 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.236420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.236536 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.236617 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.236621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.237241 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.237244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.237733 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.237739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.238166 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.238168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.238232 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.238586 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.238596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.239754 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.241014 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.241016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.241018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.241180 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.241373 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.241374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.241376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.241434 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.241727 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.241730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.241811 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.241932 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.242045 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:55:04.242209 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.236149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.0.98.61:38155)
0620 08:55:04.236217 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.236219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.236230 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:04.236312 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.236314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.236343 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.236402 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.236692 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.237155 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.237853 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.238085 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.238167 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.239852 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.239855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.239857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.239876 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.240922 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.241252 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.241254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.241255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.241316 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.241493 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.241496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.241497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.241628 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.241694 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.241964 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.242095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":22}
I20260620 08:55:04.245215 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Bootstrap starting.
I20260620 08:55:04.247380 7041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:04.248183 7041 log.cc:824] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:04.249977 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: No bootstrap required, opened a new log
I20260620 08:55:04.252898 7041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } }
I20260620 08:55:04.253089 7041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:04.253126 7041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f27f25fbfe44a4d949940405fa61719, State: Initialized, Role: FOLLOWER
I20260620 08:55:04.253515 7041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } }
I20260620 08:55:04.253927 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } } }
I20260620 08:55:04.254055 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:04.254647 7041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 08:55:04.258836 7058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:55:04.258903 7058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:55:04.260938 6973 master_runner.cc:428] Detected that this master 7f27f25fbfe44a4d949940405fa61719 is joining an existing cluster
I20260620 08:55:04.260985 6973 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38155
I20260620 08:55:04.270002 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.262820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.0.98.61:38155)
0620 08:55:04.262927 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.262932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.263380 (+ 448us) server_negotiation.cc:407] Connection header received
0620 08:55:04.263523 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.263526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.263560 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.263627 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.264169 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.264848 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.265665 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.265900 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.265991 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.267170 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.267174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.267176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.267195 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.268593 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.268949 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.268952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.268953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.269031 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.269255 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.269258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.269260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.269412 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.269489 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.269709 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.269859 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":3456}
I20260620 08:55:04.270012 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.262933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35364)
0620 08:55:04.263308 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.263323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.263428 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.263743 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.263746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.263945 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.264066 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.264073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.264934 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.264938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.265539 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.265547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.266009 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.266013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.266101 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.266581 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.266596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.267043 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.268684 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.268687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.268689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.268875 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.269109 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.269112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.269114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.269188 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.269486 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.269489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.269582 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.269664 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.269809 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":18,"thread_start_us":83,"threads_started":1}
I20260620 08:55:04.270599 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.263104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.0.98.62:36477)
0620 08:55:04.263211 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.263215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.263227 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:04.263348 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.263351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.263397 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.263462 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.263929 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.264662 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.265862 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.266154 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.266238 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.267285 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.267289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.267292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.267317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.268878 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.269362 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.269365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.269367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.269464 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.269751 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.269755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.269757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.269944 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.270007 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.270320 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.270456 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:55:04.271710 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.262790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34162)
0620 08:55:04.263167 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.263182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.263266 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.263472 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.263474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.263721 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.263825 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.263831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.264762 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.264766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.265744 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.265753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.266243 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.266246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.266328 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.266764 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.266778 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.267160 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.268997 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.269000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.269002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.269254 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.269572 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.269574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.269576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.269658 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.270072 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.270074 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.270156 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.270731 (+ 575us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.271520 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":3,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 08:55:04.285663 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.276540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.0.98.61:38155)
0620 08:55:04.276642 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.276645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.276788 (+ 143us) server_negotiation.cc:407] Connection header received
0620 08:55:04.276967 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.276970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.277018 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.277087 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.277762 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.278514 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.279793 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.280121 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.280211 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.282657 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.282661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.282664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.282687 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.284102 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.284605 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.284608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.284609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.284678 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.284913 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.284915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.284916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.285067 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.285138 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.285454 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.285554 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:04.285849 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.275878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35378)
0620 08:55:04.276675 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.276721 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.276838 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.277125 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.277130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.277532 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.277659 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.277666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.278617 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.278620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.279695 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.279706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.280228 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.280232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.280334 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.280845 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.280858 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.282519 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.284198 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.284206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.284212 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.284501 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.284754 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.284757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.284759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.284845 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.285147 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.285153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.285275 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.285422 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.285605 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":77,"threads_started":1}
I20260620 08:55:04.287166 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38155: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7f27f25fbfe44a4d949940405fa61719"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38155
}
attrs {
promote: true
}
}
dest_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7"
cas_config_opid_index: -1
I20260620 08:55:04.287827 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } }
I20260620 08:55:04.289515 7068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7f27f25fbfe44a4d949940405fa61719 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } }
I20260620 08:55:04.290091 7015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:04.291360 7068 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38155
I20260620 08:55:04.291540 7015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Refusing update from remote peer 2c3aed0df60b4a10ae10f6ab759f4ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 08:55:04.291714 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } }
I20260620 08:55:04.291808 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:04.292068 6973 master.cc:537] Master@127.0.98.61:38155 shutting down...
I20260620 08:55:04.292845 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:55:04.295719 6496 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 08:55:04.296672 6973 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 08:55:04.296772 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 -> Peer 7f27f25fbfe44a4d949940405fa61719 (127.0.98.61:38155): Couldn't send request to peer 7f27f25fbfe44a4d949940405fa61719. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:55:04.296950 6973 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:04.297035 6973 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 08:55:04.302945 6973 master.cc:552] Master@127.0.98.61:38155 shutdown complete.
I20260620 08:55:04.303004 6973 master_runner.cc:305] Clearing existing system tablet
I20260620 08:55:04.303236 6973 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.303287 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.304572 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.305424 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.305644 6973 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.305768 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.305799 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
uuid: "7f27f25fbfe44a4d949940405fa61719"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "427a64ade18606c2d15df8e3cb4951c8"
server_key_iv: "81373f9991bb258ca18eed61c21a20aa"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.307946 6973 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:04.310632 6973 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 08:55:04.310719 6973 log.cc:1196] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 08:55:04.311363 6973 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Deleting consensus metadata
I20260620 08:55:04.311532 6973 master_runner.cc:315] Copying system tablet from 127.0.98.62:36477
I20260620 08:55:04.312662 6973 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 08:55:04.314155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38523
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:55:04.321949 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.313204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34176)
0620 08:55:04.313757 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.313774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.313895 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.314159 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.314162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.314366 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.314490 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.314497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.315532 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.315536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.316562 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.316574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.317278 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.317283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.317382 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.318034 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.318051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.318482 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.320379 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.320382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.320385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.320633 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.320912 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.320915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.320917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.321000 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.321404 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.321407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.321504 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.321617 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.321748 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":91,"threads_started":1}
I20260620 08:55:04.321964 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.313298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.0.98.62:36477)
0620 08:55:04.313427 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.313432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.313860 (+ 428us) server_negotiation.cc:407] Connection header received
0620 08:55:04.313983 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.313987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.314038 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.314121 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.314599 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.315401 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.316774 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.317133 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.317236 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.318619 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.318623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.318626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.318653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.320281 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.320732 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.320736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.320737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.320824 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.321083 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.321087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.321089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.321306 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.321403 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.321685 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.321801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:55:04.322924 6529 tablet_copy_service.cc:140] P 2c3aed0df60b4a10ae10f6ab759f4ce7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7f27f25fbfe44a4d949940405fa61719 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34176)
I20260620 08:55:04.323123 6529 tablet_copy_service.cc:161] P 2c3aed0df60b4a10ae10f6ab759f4ce7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7f27f25fbfe44a4d949940405fa61719 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34176: session id = 7f27f25fbfe44a4d949940405fa61719-00000000000000000000000000000000
I20260620 08:55:04.325083 6529 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 08:55:04.326681 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 08:55:04.328946 6973 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: tablet copy: Starting download of 0 data blocks...
I20260620 08:55:04.329126 6973 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: tablet copy: Starting download of 1 WAL segments...
I20260620 08:55:04.330757 6973 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 08:55:04.332479 6529 tablet_copy_service.cc:342] P 2c3aed0df60b4a10ae10f6ab759f4ce7: Request end of tablet copy session 7f27f25fbfe44a4d949940405fa61719-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34176
I20260620 08:55:04.332590 6529 tablet_copy_service.cc:433] P 2c3aed0df60b4a10ae10f6ab759f4ce7: ending tablet copy session 7f27f25fbfe44a4d949940405fa61719-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7f27f25fbfe44a4d949940405fa61719
I20260620 08:55:04.339951 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:04.341650 7082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.341699 7085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.343735 7083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.351794 6973 server_base.cc:1034] running on GCE node
I20260620 08:55:04.352020 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:04.352344 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:04.353526 6973 hybrid_clock.cc:648] HybridClock initialized: now 1781945704353514 us; error 31 us; skew 500 ppm
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:04.355147 6973 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 08:55:04.355659 6973 webserver.cc:460] Webserver started at http://127.0.98.61:40257/ using document root <none> and password file <none>
I20260620 08:55:04.355847 6973 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.355886 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.357177 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.358116 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.358340 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.358409 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
uuid: "7f27f25fbfe44a4d949940405fa61719"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "427a64ade18606c2d15df8e3cb4951c8"
server_key_iv: "81373f9991bb258ca18eed61c21a20aa"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.358485 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.369270 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:04.371137 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:04.375743 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38155
I20260620 08:55:04.376363 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
I20260620 08:55:04.377991 7145 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:55:04.378976 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Bootstrap starting.
I20260620 08:55:04.379740 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38155 every 8 connection(s)
I20260620 08:55:04.388919 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:55:04.389209 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Bootstrap complete.
I20260620 08:55:04.389739 7145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } }
I20260620 08:55:04.389866 7145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f27f25fbfe44a4d949940405fa61719, State: Initialized, Role: LEARNER
I20260620 08:55:04.389959 7145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } }
I20260620 08:55:04.390288 7148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } }
I20260620 08:55:04.390385 7145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:04.390479 7148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:55:04.393587 7159 catalog_manager.cc:1259] Loaded cluster ID: a9e19b1412e840e9b9a52a6d69159b9a
I20260620 08:55:04.393646 7159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: loading cluster ID for follower catalog manager: success
I20260620 08:55:04.395819 7159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: acquiring CA information for follower catalog manager: success
I20260620 08:55:04.397167 7159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 08:55:04.436146 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:04.436379 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:04.436434 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:04.440342 7081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:04.440390 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:04.440415 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:04.440441 7081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:04.440470 7081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:04.445509 7081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:04.446504 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:04.447481 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:04.455227 7164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.455336 7167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.455376 7081 server_base.cc:1034] running on GCE node
W20260620 08:55:04.455521 7165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.456022 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:04.457041 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:04.458209 7081 hybrid_clock.cc:648] HybridClock initialized: now 1781945704458194 us; error 26 us; skew 500 ppm
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:04.460826 7081 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 08:55:04.461678 7081 webserver.cc:460] Webserver started at http://127.0.98.60:46711/ using document root <none> and password file <none>
I20260620 08:55:04.462186 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.462241 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.462423 7081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:04.464788 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/instance:
uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "19c00c822caf1545da7b201567851b55"
server_key_iv: "896a1c9669f6611e9ddd907a3e00f144"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.465304 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal/instance:
uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "19c00c822caf1545da7b201567851b55"
server_key_iv: "896a1c9669f6611e9ddd907a3e00f144"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.468761 7081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 08:55:04.470899 7174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.472110 7081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 08:55:04.472210 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "19c00c822caf1545da7b201567851b55"
server_key_iv: "896a1c9669f6611e9ddd907a3e00f144"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.472297 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.491125 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:04.494194 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:04.494374 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:04.501181 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38523
I20260620 08:55:04.501423 7226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38523 every 8 connection(s)
I20260620 08:55:04.502084 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
I20260620 08:55:04.504755 7227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:04.508168 7227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:55:04.511040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 7081
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 08:55:04.521278 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.509939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.62:36477)
0620 08:55:04.510076 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.510079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.510784 (+ 705us) server_negotiation.cc:407] Connection header received
0620 08:55:04.511896 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.511900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.511954 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.512008 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.513163 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.513719 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.514859 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.515210 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.515303 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.517940 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.517944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.517946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.517980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.519250 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.519773 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.519776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.519778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.519849 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.520164 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.520168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.520169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.520359 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.520455 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.520991 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.521118 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:55:04.522142 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.510217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34190)
0620 08:55:04.510644 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.510685 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.511679 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.512299 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.512305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.512620 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.513034 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.513046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.513835 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.513839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.514758 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.514765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.515375 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.515379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.515575 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.516191 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.516205 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.517759 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.519346 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.519353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.519362 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.519691 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.519939 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.519941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.519943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.520069 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.520450 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.520455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.520720 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.520893 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.521197 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":98,"threads_started":1}
I20260620 08:55:04.522142 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.511837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.0.98.60:38523)
0620 08:55:04.512152 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.512160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.512190 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:55:04.512276 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.512279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.512426 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.512570 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.513292 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.514314 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.515328 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.515841 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.515929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.518278 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.518284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.518296 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.518319 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.519812 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.520214 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.520218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.520220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.520283 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.520547 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.520550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.520552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.520802 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.520988 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.521217 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.521330 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"spinlock_wait_cycles":1920,"thread_start_us":145,"threads_started":1}
I20260620 08:55:04.523310 7227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 08:55:04.531342 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.523753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35386)
0620 08:55:04.523844 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.523857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.523926 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.524787 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.524791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.524921 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.525008 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.525013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.525858 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.525860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.526400 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.526408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.526840 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.526843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.526922 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.527383 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.527419 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.528700 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.530267 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.530269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.530270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.530452 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.530656 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.530657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.530659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.530722 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.531000 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.531002 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.531071 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.531147 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.531236 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 08:55:04.531390 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.523827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.0.98.61:38155)
0620 08:55:04.524074 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.524080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.524098 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:55:04.524147 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.524150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.524203 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.524539 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.525104 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.525747 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.526518 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.526749 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.526828 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.528839 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.528841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.528843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.528864 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.530165 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.530528 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.530530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.530531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.530598 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.530781 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.530783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.530784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.530933 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.531000 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.531186 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.531274 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260620 08:55:04.532174 7227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 08:55:04.538556 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.532574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33510)
0620 08:55:04.532668 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.532681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.532740 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.532884 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.532886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.532999 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.533063 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.533067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.533666 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.533668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.534134 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.534139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.534490 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.534493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.534550 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.534906 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.534917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.536113 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.537514 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.537517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.537518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.537684 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.537887 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.537889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.537890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.537970 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.538234 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.538236 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.538298 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.538366 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.538456 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:55:04.538673 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.532638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.98.60:38523)
0620 08:55:04.532711 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.532714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.532724 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:55:04.532792 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.532794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.532823 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.532875 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.533134 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.533594 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.534211 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.534423 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.534488 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.536235 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.536238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.536240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.536258 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.537421 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.537758 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.537761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.537762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.537825 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.538030 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.538033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.538035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.538172 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.538240 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.538395 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.538520 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":24}
I20260620 08:55:04.541551 7227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Bootstrap starting.
I20260620 08:55:04.543733 7227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:04.544392 7227 log.cc:824] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:04.546130 7227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: No bootstrap required, opened a new log
I20260620 08:55:04.549167 7227 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } }
I20260620 08:55:04.549369 7227 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:04.549423 7227 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00eb79d59d4c4c9ab8e617b48f1304, State: Initialized, Role: FOLLOWER
I20260620 08:55:04.549803 7227 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } }
I20260620 08:55:04.550210 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } } }
I20260620 08:55:04.550328 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:04.550899 7227 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 08:55:04.555284 7244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:55:04.555370 7244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:55:04.558593 7081 master_runner.cc:428] Detected that this master bc00eb79d59d4c4c9ab8e617b48f1304 is joining an existing cluster
I20260620 08:55:04.558653 7081 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38523
I20260620 08:55:04.576068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6571
I20260620 08:55:04.582892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46045
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36477
--webserver_port=43093
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:55:04.589711 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.560654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34206)
0620 08:55:04.560934 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.560948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.561043 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.561386 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.561389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.561620 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.561755 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.561763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.562808 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.562811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.564038 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.564045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.564682 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.564686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.564882 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.565432 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.565446 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.565936 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.567444 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.567447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.567449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.567726 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.567989 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.567993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.567995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.588044 (+ 20049us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.588995 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.588998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.589112 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.589195 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.589352 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":81,"spinlock_wait_cycles":6272,"thread_start_us":84,"threads_started":1}
I20260620 08:55:04.589941 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.560793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.0.98.62:36477)
0620 08:55:04.560944 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.560948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.561006 (+ 58us) server_negotiation.cc:407] Connection header received
0620 08:55:04.561137 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.561140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.561185 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.561252 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.561870 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.562686 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.564143 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.564457 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.564519 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.566065 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.566069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.566072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.566098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.567344 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.567821 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.567825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.567827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.567905 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.588370 (+ 20465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.588375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.588377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.588663 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.588766 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.589653 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.589800 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 08:55:04.593431 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.560755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.0.98.61:38155)
0620 08:55:04.560869 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.560873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.561391 (+ 518us) server_negotiation.cc:407] Connection header received
0620 08:55:04.561550 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.561554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.561602 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.561649 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.562190 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.562921 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.564113 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.564449 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.564519 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.589130 (+ 24611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.589135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.589137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.589170 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.591267 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.591809 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.591812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.591814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.591902 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.592312 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.592316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.592317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.592581 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.592678 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.593097 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.593253 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:55:04.594002 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.561064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35394)
0620 08:55:04.561321 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.561334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.561449 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.561769 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.561771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.561947 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.562065 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.562072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.563032 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.563034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.563981 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.563991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.564682 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.564685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.564886 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.588546 (+ 23660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.588562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.588962 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.591404 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.591406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.591409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.591697 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.592007 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.592010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.592011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.592177 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.592756 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.592759 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.592926 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.593669 (+ 743us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.593832 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":163,"spinlock_wait_cycles":1280,"thread_start_us":105,"threads_started":1}
I20260620 08:55:04.595556 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.561434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.0.98.60:38523)
0620 08:55:04.561522 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.561526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.561891 (+ 365us) server_negotiation.cc:407] Connection header received
0620 08:55:04.562037 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.562040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.562074 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.562116 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.562756 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.563493 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.565244 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.565517 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.565615 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.567680 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.567684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.567686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.567717 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.568929 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.588816 (+ 19887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.588820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.588821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.588899 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.593051 (+ 4152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.593054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.593056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.593241 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.593320 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.595303 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.595411 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":27,"spinlock_wait_cycles":3328}
I20260620 08:55:04.596312 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.561548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33514)
0620 08:55:04.561817 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.561829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.561952 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.562223 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.562226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.562421 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.562546 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.562553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.563635 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.563638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.565137 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.565147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.565614 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.565617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.565696 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.566338 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.566353 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.567560 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.569018 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.569021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.569023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.588386 (+ 19363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.590421 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.590425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.590427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.592924 (+ 2497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.595056 (+ 2132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.595059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.595148 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.595996 (+ 848us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.596149 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945703, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 08:55:04.603391 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.595168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33520)
0620 08:55:04.595421 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.595434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.595544 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.596115 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.596119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.596288 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.596394 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.596403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.597503 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.597506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.598147 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.598156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.598579 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.598583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.598672 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.599075 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.599087 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.600439 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.602034 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.602037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.602039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.602228 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.602465 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.602468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.602470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.602548 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.602901 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.602904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.602998 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.603081 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.603230 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 08:55:04.603390 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.595690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.0.98.60:38523)
0620 08:55:04.595802 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.595806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.595817 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:04.595879 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.595882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.595926 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.595991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.596527 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.597214 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.598261 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.598474 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.598559 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.600572 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.600575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.600577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.600596 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.601930 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.602312 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.602315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.602317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.602387 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.602650 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.602653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.602655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.602815 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.602898 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.603140 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.603234 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:55:04.604885 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38523: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38523
}
attrs {
promote: true
}
}
dest_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7"
cas_config_opid_index: 5
I20260620 08:55:04.605423 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } }
I20260620 08:55:04.606400 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.607503 7069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38523
I20260620 08:55:04.607717 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.607806 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:04.608739 7081 master.cc:537] Master@127.0.98.60:38523 shutting down...
I20260620 08:55:04.608706 7201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:04.610093 7201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Refusing update from remote peer 2c3aed0df60b4a10ae10f6ab759f4ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 08:55:04.610771 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:55:04.613232 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 -> Peer bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60:38523): Couldn't send request to peer bc00eb79d59d4c4c9ab8e617b48f1304. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:55:04.614606 7081 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:04.614861 7081 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:04.614941 7081 tablet_replica.cc:330] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: stopping tablet replica
I20260620 08:55:04.617189 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.608513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.98.61:38155)
0620 08:55:04.612234 (+ 3721us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.612240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.612567 (+ 327us) server_negotiation.cc:407] Connection header received
0620 08:55:04.612621 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.612624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.612685 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.612765 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:04.613103 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.613828 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.615428 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.616056 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.616136 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.616294 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.616978 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.617056 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3624}
I20260620 08:55:04.617475 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.608280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35398)
0620 08:55:04.612387 (+ 4107us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.612401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.612478 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.612887 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.612890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.612906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:04.612990 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.612997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.613951 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.613955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.615292 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.615302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.616388 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.616391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.616478 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.616523 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.616603 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.616676 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4039}
I20260620 08:55:04.618007 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 LEARNER]: Refusing update from remote peer 2c3aed0df60b4a10ae10f6ab759f4ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 08:55:04.618430 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 08:55:04.620419 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.622036 7081 master.cc:552] Master@127.0.98.60:38523 shutdown complete.
I20260620 08:55:04.622090 7081 master_runner.cc:305] Clearing existing system tablet
I20260620 08:55:04.622047 7148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c3aed0df60b4a10ae10f6ab759f4ce7. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.622149 7148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:55:04.622228 7148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: true } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.622315 7148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:55:04.622318 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.622361 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.623370 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.623626 7068 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: attempting to promote NON_VOTER 7f27f25fbfe44a4d949940405fa61719 to VOTER
I20260620 08:55:04.624094 7068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } }
I20260620 08:55:04.625463 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.625222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33528)
0620 08:55:04.625325 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.625392 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 08:55:04.625629 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 LEARNER]: Refusing update from remote peer 2c3aed0df60b4a10ae10f6ab759f4ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 08:55:04.626276 7069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 08:55:04.626368 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.626177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33544)
0620 08:55:04.626266 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.626312 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:55:04.626459 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 -> Peer bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60:38523): Couldn't send request to peer bc00eb79d59d4c4c9ab8e617b48f1304. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 08:55:04.626740 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.626958 7081 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.627090 7081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 08:55:04.627125 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "19c00c822caf1545da7b201567851b55"
server_key_iv: "896a1c9669f6611e9ddd907a3e00f144"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.628773 7081 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:04.630787 7081 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 08:55:04.630868 7081 log.cc:1196] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 08:55:04.631115 7081 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Deleting consensus metadata
I20260620 08:55:04.631274 7081 master_runner.cc:315] Copying system tablet from 127.0.98.62:36477
I20260620 08:55:04.631345 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7f27f25fbfe44a4d949940405fa61719 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.632423 7081 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36477
I20260620 08:55:04.632822 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.632920 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:04.633431 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.633533 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:04.633946 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7f27f25fbfe44a4d949940405fa61719 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.635458 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.635576 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:04.641711 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.634163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.0.98.62:36477)
0620 08:55:04.634291 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.634295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.634809 (+ 514us) server_negotiation.cc:407] Connection header received
0620 08:55:04.634982 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.634985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.635031 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.635094 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.636072 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.636588 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.637367 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.637584 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.637666 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.638841 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.638845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.638847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.638872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.640272 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.640646 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.640649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.640651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.640727 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.640959 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.640962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.640964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.641137 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.641217 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.641450 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.641547 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 08:55:04.641731 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.634497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34208)
0620 08:55:04.634735 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.634749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.634881 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.635685 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.635692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.635875 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.635960 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.635966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.636691 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.636695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.637265 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.637272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.637681 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.637684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.637772 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.638253 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.638269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.638715 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.640361 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.640364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.640366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.640562 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.640802 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.640805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.640806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.640884 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.641218 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.641221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.641310 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.641394 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.641545 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":53,"threads_started":1}
I20260620 08:55:04.642331 6529 tablet_copy_service.cc:140] P 2c3aed0df60b4a10ae10f6ab759f4ce7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bc00eb79d59d4c4c9ab8e617b48f1304 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34208)
I20260620 08:55:04.642417 6529 tablet_copy_service.cc:161] P 2c3aed0df60b4a10ae10f6ab759f4ce7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bc00eb79d59d4c4c9ab8e617b48f1304 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34208: session id = bc00eb79d59d4c4c9ab8e617b48f1304-00000000000000000000000000000000
I20260620 08:55:04.643196 6529 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 08:55:04.644294 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 08:55:04.646219 7081 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: tablet copy: Starting download of 0 data blocks...
I20260620 08:55:04.646375 7081 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: tablet copy: Starting download of 1 WAL segments...
I20260620 08:55:04.647655 7081 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 08:55:04.649161 6529 tablet_copy_service.cc:342] P 2c3aed0df60b4a10ae10f6ab759f4ce7: Request end of tablet copy session bc00eb79d59d4c4c9ab8e617b48f1304-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34208
I20260620 08:55:04.649252 6529 tablet_copy_service.cc:433] P 2c3aed0df60b4a10ae10f6ab759f4ce7: ending tablet copy session bc00eb79d59d4c4c9ab8e617b48f1304-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bc00eb79d59d4c4c9ab8e617b48f1304
I20260620 08:55:04.658119 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:04.660175 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.660184 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.660364 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.660614 7081 server_base.cc:1034] running on GCE node
I20260620 08:55:04.660758 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:04.661017 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:04.662166 7081 hybrid_clock.cc:648] HybridClock initialized: now 1781945704662157 us; error 26 us; skew 500 ppm
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:04.663506 7081 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 08:55:04.663951 7081 webserver.cc:460] Webserver started at http://127.0.98.60:40989/ using document root <none> and password file <none>
I20260620 08:55:04.664126 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.664160 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.665227 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.666033 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.666239 7081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:04.666306 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
format_stamp: "Formatted at 2026-06-20 08:55:04 on dist-test-slave-fb1v"
server_key: "19c00c822caf1545da7b201567851b55"
server_key_iv: "896a1c9669f6611e9ddd907a3e00f144"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.666390 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.673623 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:04.675360 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:04.679574 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38523
I20260620 08:55:04.680157 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
I20260620 08:55:04.681886 7329 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:55:04.682914 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Bootstrap starting.
I20260620 08:55:04.683754 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38523 every 8 connection(s)
I20260620 08:55:04.693825 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:55:04.694155 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: Bootstrap complete.
I20260620 08:55:04.694764 7329 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } }
I20260620 08:55:04.694906 7329 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc00eb79d59d4c4c9ab8e617b48f1304, State: Initialized, Role: LEARNER
I20260620 08:55:04.695019 7329 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } }
I20260620 08:55:04.695353 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:04.695565 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:55:04.695477 7329 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:04.698341 7343 catalog_manager.cc:1259] Loaded cluster ID: a9e19b1412e840e9b9a52a6d69159b9a
I20260620 08:55:04.698441 7343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: loading cluster ID for follower catalog manager: success
I20260620 08:55:04.700754 7343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: acquiring CA information for follower catalog manager: success
I20260620 08:55:04.702147 7343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 08:55:04.724004 7253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:04.724257 7253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:04.724299 7253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:04.728524 7253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:04.728602 7253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:04.728677 7253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:04.734107 7253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43093
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:04.735168 7253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:04.736191 7253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:04.745721 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.745772 7253 server_base.cc:1034] running on GCE node
W20260620 08:55:04.745716 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.745721 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.746390 7253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:04.747074 7253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:04.748313 7253 hybrid_clock.cc:648] HybridClock initialized: now 1781945704748264 us; error 42 us; skew 500 ppm
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:04.752120 7253 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 08:55:04.753050 7253 webserver.cc:460] Webserver started at http://127.0.98.1:43093/ using document root <none> and password file <none>
I20260620 08:55:04.753615 7253 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.753698 7253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.757586 7253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:55:04.759632 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.760921 7253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:04.761039 7253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "dbb7ef58462741c6bde02a0a9c023fa9"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "66e2ee2f56118ca77f41f383b7bd6882"
server_key_iv: "e6d91270839ce013c19bc9e90f5da3bb"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.761600 7253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.782542 7253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:04.785710 7253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:04.785883 7253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:04.786439 7253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:04.787261 7253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:04.787308 7253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.787359 7253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:04.787387 7253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.798904 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46045 every 8 connection(s)
I20260620 08:55:04.798905 7253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46045
I20260620 08:55:04.800421 7253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:04.801039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 7253
I20260620 08:55:04.801237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6705
I20260620 08:55:04.808120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39981
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36477
--webserver_port=38479
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 08:55:04.818130 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.804911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49475 (local address 127.0.98.61:38155)
0620 08:55:04.805022 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.805026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.805454 (+ 428us) server_negotiation.cc:407] Connection header received
0620 08:55:04.805613 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.805616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.805665 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.805712 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.807166 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.807982 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.809724 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.810049 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.810120 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.814581 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.814585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.814588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.814617 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.815946 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816416 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.816419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.816421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.816490 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816855 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.816858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.816860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.817066 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.817135 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.817841 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.817965 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:55:04.818133 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.804067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58547 (local address 127.0.98.60:38523)
0620 08:55:04.804281 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.804285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.804579 (+ 294us) server_negotiation.cc:407] Connection header received
0620 08:55:04.805178 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.805182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.805233 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.805294 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.807186 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.808359 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.809547 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.809867 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.809953 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.813048 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.813052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.813055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.813082 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.814949 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816382 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.816385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.816387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.816483 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816856 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.816859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.816861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.817066 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.817140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.817841 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.817966 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260620 08:55:04.818200 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.802449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53027 (local address 127.0.98.62:36477)
0620 08:55:04.802614 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.802619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.803883 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 08:55:04.805067 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.805070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.805123 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.805173 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.807041 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.807795 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.809771 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.810106 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.810176 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.813580 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:04.813584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:04.813587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:04.813614 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:04.815681 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816371 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.816375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.816377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.816464 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816873 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:04.816877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:04.816879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:04.817110 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:04.817181 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:04.817924 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 08:55:04.818044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:55:04.819877 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.804810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.98.1:49475)
0620 08:55:04.805383 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.805395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.805506 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.805827 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.805829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.806616 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.807068 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.807073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.808110 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.808114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.809620 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.809625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.810267 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.810270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.810357 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.810979 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.810998 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.814420 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.816042 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.816046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.816319 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.816593 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.816598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.816753 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.817201 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.817208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.817566 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.818398 (+ 832us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.818761 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":687,"spinlock_wait_cycles":12416,"thread_start_us":70,"threads_started":1}
I20260620 08:55:04.819888 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.802834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.1:53027)
0620 08:55:04.803725 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.803778 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.804820 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.805547 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.805554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.806829 (+ 1275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.806917 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.806922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.807948 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.807951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.809648 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.809658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.810322 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.810325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.810400 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.811036 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.811052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.813455 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.815801 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.815804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.815806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.816257 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.816576 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.816581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.816760 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.817252 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.817255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.817583 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.818356 (+ 773us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.818852 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":677,"spinlock_wait_cycles":18176,"thread_start_us":503,"threads_started":1}
I20260620 08:55:04.820556 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.804235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.98.1:58547)
0620 08:55:04.804499 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.804513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.805056 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.805545 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.805554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.806209 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.806890 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.806910 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.808471 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.808474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.809435 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.809445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.810109 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.810114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.810304 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.811113 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.811125 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.812872 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:04.815288 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.815293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.815308 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.816272 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.816593 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:04.816596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:04.816598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:04.816753 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:04.817201 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:04.817208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:04.817566 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 08:55:04.818766 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0620 08:55:04.818905 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":155,"spinlock_wait_cycles":63488,"thread_start_us":111,"threads_started":1}
I20260620 08:55:04.821091 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36477
I20260620 08:55:04.821321 7472 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:04.821655 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38523
I20260620 08:55:04.821734 7473 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:04.821866 7472 heartbeater.cc:511] Master 127.0.98.62:36477 requested a full tablet report, sending...
I20260620 08:55:04.821880 7473 heartbeater.cc:511] Master 127.0.98.60:38523 requested a full tablet report, sending...
I20260620 08:55:04.822345 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38155
I20260620 08:55:04.822419 7474 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:04.822573 7474 heartbeater.cc:511] Master 127.0.98.61:38155 requested a full tablet report, sending...
I20260620 08:55:04.823036 6509 ts_manager.cc:194] Re-registered known tserver with Master: dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045)
I20260620 08:55:04.823318 7293 ts_manager.cc:194] Registered new tserver with Master: dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045)
I20260620 08:55:04.823556 7109 ts_manager.cc:194] Registered new tserver with Master: dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045)
I20260620 08:55:04.823889 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53027
W20260620 08:55:04.921183 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:04.921444 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:04.921490 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:04.925474 7480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:04.925550 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:04.925633 7480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:04.930886 7480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38479
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:04.931893 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:04.932813 7480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:04.942075 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:04.942066 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.942210 7480 server_base.cc:1034] running on GCE node
W20260620 08:55:04.942068 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:04.942719 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:04.943286 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:04.944464 7480 hybrid_clock.cc:648] HybridClock initialized: now 1781945704944425 us; error 74 us; skew 500 ppm
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:04.947163 7480 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 08:55:04.948001 7480 webserver.cc:460] Webserver started at http://127.0.98.2:38479/ using document root <none> and password file <none>
I20260620 08:55:04.948498 7480 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:04.948568 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:04.952243 7480 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:04.954435 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.956127 7480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:04.956240 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "d8b39082886f433c9b1a786b1c8e4328"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "9f7c1806178bb1313c9e5adcae39bce2"
server_key_iv: "489df795bb3c43f9beb05cae556f70ba"
server_key_version: "encryptionkey@0"
I20260620 08:55:04.956784 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:04.972869 7480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:04.975937 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:04.976114 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:04.976706 7480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:04.977577 7480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:04.977623 7480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.977662 7480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:04.977676 7480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:04.986541 7480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39981
I20260620 08:55:04.986585 7608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39981 every 8 connection(s)
I20260620 08:55:04.987516 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:04.994611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 7480
I20260620 08:55:04.994800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6839
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 08:55:04 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 08:55:05.003424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45053
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36477
--webserver_port=39373
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:55:05.004215 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.990770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39569 (local address 127.0.98.61:38155)
0620 08:55:04.990882 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.990904 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.992147 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 08:55:04.992287 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.992289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.992332 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.992397 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.993480 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.994280 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.995927 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.996259 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.996332 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.000886 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.000890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.000893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.000933 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.002343 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.002799 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.002802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.002816 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.002899 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.003189 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.003192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.003194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.003401 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.003479 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.003929 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.004056 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:55:05.004312 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.989408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42015 (local address 127.0.98.60:38523)
0620 08:55:04.989545 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.989549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.990508 (+ 959us) server_negotiation.cc:407] Connection header received
0620 08:55:04.991893 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.991897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.991953 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.992047 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.993500 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.994280 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.995938 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.996290 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.996350 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.000140 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.000144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.000147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.000177 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.001776 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.002372 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.002376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.002378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.002460 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.002826 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.002829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.002831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.003055 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.003130 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.004056 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.004173 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:55:05.004391 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.989379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51857 (local address 127.0.98.62:36477)
0620 08:55:04.989525 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:04.989530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:04.991413 (+ 1883us) server_negotiation.cc:407] Connection header received
0620 08:55:04.991893 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:04.991897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:04.991965 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:04.992047 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:04.993476 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.994036 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.995941 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.996272 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.996356 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.000625 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.000629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.000631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.000657 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.002683 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.003122 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.003126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.003128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.003208 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.003483 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.003486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.003488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.003715 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.003802 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.004121 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.004227 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 08:55:05.005915 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.990673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.98.2:39569)
0620 08:55:04.992073 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.992086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.992204 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.992436 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.992438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.992758 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.993323 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.993342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.994426 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.994429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.995802 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.995812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.996525 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.996529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.996813 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.997579 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.997594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:05.000718 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:05.002457 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.002460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.002462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.002692 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.003000 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.003003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.003005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.003096 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.003532 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:05.003534 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:05.003652 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.004391 (+ 739us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.004813 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"mutex_wait_us":314,"spinlock_wait_cycles":40960,"thread_start_us":91,"threads_started":1}
I20260620 08:55:05.005915 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.989750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.98.2:42015)
0620 08:55:04.990328 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.990399 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.991666 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.992608 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.992611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.992789 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.993333 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.993342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.994426 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.994429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.995805 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.995811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.996525 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.996529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.996808 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.997682 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.997695 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:04.999977 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:05.001929 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.001935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.001950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.002259 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.002558 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.002560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.002562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.002732 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.003189 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:05.003195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:05.003496 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.004930 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.005048 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":293,"spinlock_wait_cycles":1408,"thread_start_us":127,"threads_started":1}
I20260620 08:55:05.006654 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:04.989751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:51857)
0620 08:55:04.990797 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:04.990812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:04.991666 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:04.992227 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:04.992236 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:04.992779 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:04.993327 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.993341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.994169 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.994173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.995815 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:04.995825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:04.996525 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:04.996529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:04.996939 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:04.997546 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:04.997575 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:05.000473 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:05.002800 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.002803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.002805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.003018 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.003307 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.003310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.003312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.003387 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.003854 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:05.003856 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:05.003951 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.004511 (+ 560us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.004894 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":375,"spinlock_wait_cycles":26880,"thread_start_us":662,"threads_started":1}
I20260620 08:55:05.007524 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38523
I20260620 08:55:05.007762 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38155
I20260620 08:55:05.007767 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36477
I20260620 08:55:05.007794 7610 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:05.007864 7609 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:05.007864 7611 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:05.008473 7609 heartbeater.cc:511] Master 127.0.98.62:36477 requested a full tablet report, sending...
I20260620 08:55:05.008486 7610 heartbeater.cc:511] Master 127.0.98.60:38523 requested a full tablet report, sending...
I20260620 08:55:05.008473 7611 heartbeater.cc:511] Master 127.0.98.61:38155 requested a full tablet report, sending...
I20260620 08:55:05.009941 6509 ts_manager.cc:194] Re-registered known tserver with Master: d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981)
I20260620 08:55:05.010089 7293 ts_manager.cc:194] Registered new tserver with Master: d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981)
I20260620 08:55:05.010913 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51857
I20260620 08:55:05.011651 7109 ts_manager.cc:194] Registered new tserver with Master: d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981)
W20260620 08:55:05.116248 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:05.116505 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:05.116552 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:05.120610 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:05.120677 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:05.120743 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:05.126240 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39373
--tserver_master_addrs=127.0.98.62:36477,127.0.98.61:38155,127.0.98.60:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:05.127318 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:05.128346 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:05.137926 7622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:05.137917 7624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:05.137941 7621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:05.138083 7616 server_base.cc:1034] running on GCE node
I20260620 08:55:05.138931 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:05.139566 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:05.140763 7616 hybrid_clock.cc:648] HybridClock initialized: now 1781945705140727 us; error 48 us; skew 500 ppm
Jun 20 08:55:05 dist-test-slave-fb1v krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:05.144070 7616 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 08:55:05.144914 7616 webserver.cc:460] Webserver started at http://127.0.98.3:39373/ using document root <none> and password file <none>
I20260620 08:55:05.145443 7616 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:05.145524 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:05.149340 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:05.151289 7631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:05.152498 7616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:05.152607 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "eee75217f62c468dba0fcc893ec81db7"
format_stamp: "Formatted at 2026-06-20 08:55:03 on dist-test-slave-fb1v"
server_key: "8c501c9a576d9366a0afa46730282a0f"
server_key_iv: "a718fdf5aedb2e90a0d1f98aa7e255ec"
server_key_version: "encryptionkey@0"
I20260620 08:55:05.153117 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:05.169730 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.165793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33548)
0620 08:55:05.166119 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.166131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.166231 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.166606 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.166610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.166631 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:05.166730 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.166735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.167736 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.167739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.168526 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.168535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.169365 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.169368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.169437 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.169461 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.169508 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.169597 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":63,"threads_started":1}
I20260620 08:55:05.169724 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.165941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.98.60:38523)
0620 08:55:05.166068 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.166072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.166209 (+ 137us) server_negotiation.cc:407] Connection header received
0620 08:55:05.166331 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.166335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.166409 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.166520 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:05.166818 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.167629 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.168637 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.169245 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.169336 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.169487 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.169529 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.169586 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:55:05.170332 7303 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 08:55:05.170737 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c3aed0df60b4a10ae10f6ab759f4ce7. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:05.170893 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:55:05.173682 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:05.176301 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:05.176466 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:05.177021 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:05.177786 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:05.177830 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:05.177867 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:05.177881 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:05.185892 7616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45053
I20260620 08:55:05.186002 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45053 every 8 connection(s)
I20260620 08:55:05.186832 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:05.191123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 7616
Jun 20 08:55:05 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 08:55:05 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 08:55:05 dist-test-slave-fb1v krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:05.203727 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.188545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52983 (local address 127.0.98.60:38523)
0620 08:55:05.188748 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.188751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.189805 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 08:55:05.191131 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.191134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.191181 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.191251 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.193283 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.194024 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195108 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.195416 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195495 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.199213 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.199217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.199219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.199243 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.201196 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.201787 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.201790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.201792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.201871 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.202247 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.202250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.202252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.202445 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.202524 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.203408 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.203519 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260620 08:55:05.203846 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.188538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59433 (local address 127.0.98.62:36477)
0620 08:55:05.188667 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.188671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.189534 (+ 863us) server_negotiation.cc:407] Connection header received
0620 08:55:05.190734 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.190838 (+ 104us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.190906 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.190989 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.192877 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.193613 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.194842 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.195181 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195254 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.200408 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.200412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.200414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.200440 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.202125 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.202560 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.202564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.202565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.202666 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.202979 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.202982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.202984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.203173 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.203232 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.203592 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.203718 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:55:05.205076 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.188957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:59433)
0620 08:55:05.189394 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.189443 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.190441 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.191177 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.191185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.192629 (+ 1444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:05.192752 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.192759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.193734 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.193737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.194717 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.194727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.195447 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.195666 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.196390 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:05.196409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:05.200281 (+ 3872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:05.202232 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.202235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.202236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.202438 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.202800 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.202803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.202805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.202884 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.203295 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:05.203298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:05.203408 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.204339 (+ 931us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.204465 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260620 08:55:05.205076 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.188957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.98.3:52983)
0620 08:55:05.189735 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.189747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.190441 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.192863 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.192867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.193018 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:05.193145 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.193152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.194130 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.194133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.194972 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.194992 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.195637 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.195723 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.196436 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:05.196451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:05.199037 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:05.201318 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.201324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.201341 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.201677 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.201967 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.201970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.201972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.202154 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.202569 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:05.202576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:05.202920 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.203127 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.204064 (+ 937us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":582,"thread_start_us":107,"threads_started":1}
I20260620 08:55:05.206666 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36477
I20260620 08:55:05.206804 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.189884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50369 (local address 127.0.98.61:38155)
0620 08:55:05.189995 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.189999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.190104 (+ 105us) server_negotiation.cc:407] Connection header received
0620 08:55:05.190878 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.190881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.190951 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.191028 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.192492 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.193126 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195230 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.195554 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.202355 (+ 6714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.202358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.202361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.202389 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.204009 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.204982 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.204986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.204988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.205063 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.205355 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.205359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.205361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.205563 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.205629 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.206556 (+ 927us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.206666 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:55:05.206888 7747 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:05.206919 7748 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38523
I20260620 08:55:05.206976 7748 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:05.207491 7747 heartbeater.cc:511] Master 127.0.98.62:36477 requested a full tablet report, sending...
I20260620 08:55:05.207782 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.189765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.98.3:50369)
0620 08:55:05.190023 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.190035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.190474 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.191177 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.191185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.191699 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:05.192358 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.192378 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.193279 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.193282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.195095 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.195103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.195836 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.195839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.196471 (+ 632us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.196949 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:05.196964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:05.201278 (+ 4314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:05.204622 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.204625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.204627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.204874 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.205172 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:05.205175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:05.205177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:05.205258 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:05.205952 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:05.205954 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:05.206051 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.207355 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.207600 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260620 08:55:05.208133 7748 heartbeater.cc:511] Master 127.0.98.60:38523 requested a full tablet report, sending...
I20260620 08:55:05.208810 6509 ts_manager.cc:194] Re-registered known tserver with Master: eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053)
I20260620 08:55:05.208870 7293 ts_manager.cc:194] Registered new tserver with Master: eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053)
I20260620 08:55:05.209542 7749 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38155
I20260620 08:55:05.209543 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59433
I20260620 08:55:05.209630 7749 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:05.209789 7749 heartbeater.cc:511] Master 127.0.98.61:38155 requested a full tablet report, sending...
I20260620 08:55:05.210155 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.193666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.98.60:38523)
0620 08:55:05.194940 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.194953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.195535 (+ 582us) server_negotiation.cc:407] Connection header received
0620 08:55:05.196120 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.196124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.196174 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.196294 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.197264 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.197866 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.199160 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.199691 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.199783 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.206776 (+ 6993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.206780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.206782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.206807 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.208440 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.208845 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.208849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.208850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.208924 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.209205 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.209208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.209209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.209391 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.209501 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.209877 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.209999 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":72,"threads_started":1}
I20260620 08:55:05.210155 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.193048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.0.98.61:38155)
0620 08:55:05.196079 (+ 3031us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.196083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.196094 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:05.196162 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.196164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.196207 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.196283 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.201439 (+ 5156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.202203 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.203790 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.204147 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.204227 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.205707 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.205710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.205713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.205737 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.208011 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.208507 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.208510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.208512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.208591 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.208846 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.208849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.208851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.209137 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.209246 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.209909 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.210010 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2930,"thread_start_us":75,"threads_started":1}
I20260620 08:55:05.210456 7109 ts_manager.cc:194] Registered new tserver with Master: eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053)
I20260620 08:55:05.212805 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.194411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.98.62:36477)
0620 08:55:05.195065 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.195069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.195855 (+ 786us) server_negotiation.cc:407] Connection header received
0620 08:55:05.196166 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.196170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.196242 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.196316 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.197205 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.198236 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.199304 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.199669 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.199744 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.201296 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.201300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.201303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.201326 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.204515 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.211326 (+ 6811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.211330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.211332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.211428 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.211854 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.211859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.211861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.212171 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.212326 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.212559 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.212673 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":73,"threads_started":1}
I20260620 08:55:05.215041 6509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:55:05.217943 6509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:05.220491 7636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: true } } }
I20260620 08:55:05.220621 7636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:05.223637 7636 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: attempting to promote NON_VOTER bc00eb79d59d4c4c9ab8e617b48f1304 to VOTER
I20260620 08:55:05.223920 7636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } }
I20260620 08:55:05.225069 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Refusing update from remote peer 2c3aed0df60b4a10ae10f6ab759f4ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 08:55:05.225433 7765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 08:55:05.225535 7303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 LEARNER]: Refusing update from remote peer 2c3aed0df60b4a10ae10f6ab759f4ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 08:55:05.226430 7765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 08:55:05.227036 7766 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.227869 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.228494 7636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.228466 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.228587 7636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:05.228616 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:05.229499 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.229593 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:05.229882 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.230001 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:05.229912 7303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.231568 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:05.231699 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:05.237439 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.232035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45053 (local address 127.0.0.1:37096)
0620 08:55:05.232925 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.232940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.233017 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.234009 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.234012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.234029 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:05.234128 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.234135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.235061 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.235065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.236230 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.236240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.237084 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.237088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.237168 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.237190 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.237247 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.237317 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827}
I20260620 08:55:05.237493 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.232196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.0.98.3:45053)
0620 08:55:05.233584 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.233599 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.233623 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:55:05.233702 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.233709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.233890 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.234038 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:05.234236 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.234948 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.236361 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.236977 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.237074 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.237221 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.237271 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.237359 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":123,"threads_started":1}
I20260620 08:55:05.239012 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.232693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39981 (local address 127.0.0.1:36346)
0620 08:55:05.234429 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.234437 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.234481 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.234974 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.234977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.234992 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:05.235069 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.235075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.236541 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.236544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.237711 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.237721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.238636 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.238640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.238722 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.238743 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.238815 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.238885 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1698,"thread_start_us":85,"threads_started":1}
I20260620 08:55:05.239050 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.233195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.0.98.2:39981)
0620 08:55:05.234652 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.234659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.234676 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:55:05.234729 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.234733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.234863 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.234980 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:05.235175 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.236414 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.237826 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.238529 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.238624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.238747 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.238840 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.238918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"thread_start_us":96,"threads_started":1}
I20260620 08:55:05.239677 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.232389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46045 (local address 127.0.0.1:49262)
0620 08:55:05.234239 (+ 1850us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.234251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.234313 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.235559 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.235562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.235577 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:05.235685 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.235691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.236930 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.236934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.238146 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.238154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.239319 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.239323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.239407 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.239428 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.239475 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.239535 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1790,"thread_start_us":117,"threads_started":1}
I20260620 08:55:05.240342 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.232569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.0.98.1:46045)
0620 08:55:05.235238 (+ 2669us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.235246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.235262 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:55:05.235316 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.235319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.235440 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.235558 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:05.235765 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.236807 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.238254 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.238963 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.239952 (+ 989us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.240089 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.240135 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.240212 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2580,"thread_start_us":109,"threads_started":1}
I20260620 08:55:05.240669 7681 tablet_service.cc:1467] Processing CreateTablet for tablet 52af58902da24c02baada9b5fcc88e6d (DEFAULT_TABLE table=test-table [id=e0857b6de0b64cb6919f779b7bcb1964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:05.241775 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52af58902da24c02baada9b5fcc88e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:05.242376 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 52af58902da24c02baada9b5fcc88e6d (DEFAULT_TABLE table=test-table [id=e0857b6de0b64cb6919f779b7bcb1964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:05.242981 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 52af58902da24c02baada9b5fcc88e6d (DEFAULT_TABLE table=test-table [id=e0857b6de0b64cb6919f779b7bcb1964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:05.243382 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52af58902da24c02baada9b5fcc88e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:05.243927 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52af58902da24c02baada9b5fcc88e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:05.249244 7780 tablet_bootstrap.cc:492] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Bootstrap starting.
I20260620 08:55:05.250900 7781 tablet_bootstrap.cc:492] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Bootstrap starting.
I20260620 08:55:05.251302 7782 tablet_bootstrap.cc:492] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Bootstrap starting.
I20260620 08:55:05.251370 7780 tablet_bootstrap.cc:654] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:05.252871 7780 log.cc:824] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:05.253440 7781 tablet_bootstrap.cc:654] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:05.253501 7782 tablet_bootstrap.cc:654] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:05.254209 7781 log.cc:824] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:05.254266 7782 log.cc:824] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:05.255115 7780 tablet_bootstrap.cc:492] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: No bootstrap required, opened a new log
I20260620 08:55:05.255316 7780 ts_tablet_manager.cc:1397] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:55:05.256114 7781 tablet_bootstrap.cc:492] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: No bootstrap required, opened a new log
I20260620 08:55:05.256114 7782 tablet_bootstrap.cc:492] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: No bootstrap required, opened a new log
I20260620 08:55:05.256289 7782 ts_tablet_manager.cc:1397] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:55:05.256289 7781 ts_tablet_manager.cc:1397] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 08:55:05.258987 7780 raft_consensus.cc:348] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.259208 7780 raft_consensus.cc:374] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:05.259258 7780 raft_consensus.cc:729] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee75217f62c468dba0fcc893ec81db7, State: Initialized, Role: FOLLOWER
I20260620 08:55:05.259699 7780 consensus_queue.cc:260] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.260605 7781 raft_consensus.cc:348] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.260892 7781 raft_consensus.cc:374] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:05.260948 7781 raft_consensus.cc:729] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb7ef58462741c6bde02a0a9c023fa9, State: Initialized, Role: FOLLOWER
I20260620 08:55:05.261328 7747 heartbeater.cc:503] Master 127.0.98.62:36477 was elected leader, sending a full tablet report...
I20260620 08:55:05.261370 7781 consensus_queue.cc:260] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.261890 7782 raft_consensus.cc:348] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.262127 7782 raft_consensus.cc:374] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:05.262181 7782 raft_consensus.cc:729] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b39082886f433c9b1a786b1c8e4328, State: Initialized, Role: FOLLOWER
I20260620 08:55:05.262293 7781 ts_tablet_manager.cc:1428] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 08:55:05.263492 7472 heartbeater.cc:503] Master 127.0.98.62:36477 was elected leader, sending a full tablet report...
I20260620 08:55:05.263899 7782 consensus_queue.cc:260] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.264508 7609 heartbeater.cc:503] Master 127.0.98.62:36477 was elected leader, sending a full tablet report...
I20260620 08:55:05.264775 7782 ts_tablet_manager.cc:1428] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 08:55:05.266609 7780 ts_tablet_manager.cc:1428] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260620 08:55:05.297883 7788 raft_consensus.cc:482] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:05.298094 7788 raft_consensus.cc:504] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.299146 7788 leader_election.cc:288] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045), eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053)
I20260620 08:55:05.303326 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.299747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40131 (local address 127.0.98.3:45053)
0620 08:55:05.299870 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.299875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.299893 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:55:05.300042 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.300046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.300093 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.300157 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:05.300833 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.301320 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.302400 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.302805 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.302908 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.302970 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.303140 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.303204 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:55:05.303345 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.299452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45053 (local address 127.0.98.2:40131)
0620 08:55:05.299667 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.299688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.299887 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.300183 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.300185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.300204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:05.300719 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.300726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.301419 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.301422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.302288 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.302298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.302913 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.302916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.303031 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.303050 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.303115 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.303193 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":44}
I20260620 08:55:05.303686 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.299582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46045 (local address 127.0.98.2:45999)
0620 08:55:05.300041 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:05.300057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:05.300145 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:05.300356 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:05.300359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:05.300373 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:05.300484 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.300490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.301189 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.301193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.302326 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:05.302347 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.303241 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.303245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:05.303346 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.303368 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:05.303474 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:55:05.303547 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":42}
I20260620 08:55:05.304056 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.299836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45999 (local address 127.0.98.1:46045)
0620 08:55:05.299982 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.299987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.300116 (+ 129us) server_negotiation.cc:407] Connection header received
0620 08:55:05.300230 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.300233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.300276 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.300338 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:05.300583 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.301085 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.302553 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.303045 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.303225 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.303299 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.303846 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.303949 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 08:55:05.304095 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af58902da24c02baada9b5fcc88e6d" candidate_uuid: "d8b39082886f433c9b1a786b1c8e4328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee75217f62c468dba0fcc893ec81db7" is_pre_election: true
I20260620 08:55:05.304366 7701 raft_consensus.cc:2454] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b39082886f433c9b1a786b1c8e4328 in term 0.
I20260620 08:55:05.304554 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af58902da24c02baada9b5fcc88e6d" candidate_uuid: "d8b39082886f433c9b1a786b1c8e4328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" is_pre_election: true
I20260620 08:55:05.304816 7413 raft_consensus.cc:2454] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b39082886f433c9b1a786b1c8e4328 in term 0.
I20260620 08:55:05.304818 7498 leader_election.cc:302] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8b39082886f433c9b1a786b1c8e4328, eee75217f62c468dba0fcc893ec81db7; no voters:
I20260620 08:55:05.305112 7788 raft_consensus.cc:2781] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:05.305197 7788 raft_consensus.cc:482] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:05.305240 7788 raft_consensus.cc:3037] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Advancing to term 1
W20260620 08:55:05.305645 7475 tablet.cc:2369] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:55:05.306448 7788 raft_consensus.cc:504] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.306784 7788 leader_election.cc:288] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [CANDIDATE]: Term 1 election: Requested vote from peers dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045), eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053)
I20260620 08:55:05.307113 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af58902da24c02baada9b5fcc88e6d" candidate_uuid: "d8b39082886f433c9b1a786b1c8e4328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb7ef58462741c6bde02a0a9c023fa9"
I20260620 08:55:05.307179 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af58902da24c02baada9b5fcc88e6d" candidate_uuid: "d8b39082886f433c9b1a786b1c8e4328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee75217f62c468dba0fcc893ec81db7"
I20260620 08:55:05.307240 7413 raft_consensus.cc:3037] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:05.307293 7701 raft_consensus.cc:3037] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:05.308509 7413 raft_consensus.cc:2454] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b39082886f433c9b1a786b1c8e4328 in term 1.
I20260620 08:55:05.308511 7701 raft_consensus.cc:2454] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b39082886f433c9b1a786b1c8e4328 in term 1.
I20260620 08:55:05.308789 7498 leader_election.cc:302] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8b39082886f433c9b1a786b1c8e4328, eee75217f62c468dba0fcc893ec81db7; no voters:
I20260620 08:55:05.309033 7788 raft_consensus.cc:2781] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:05.309212 7788 raft_consensus.cc:686] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 1 LEADER]: Becoming Leader. State: Replica: d8b39082886f433c9b1a786b1c8e4328, State: Running, Role: LEADER
I20260620 08:55:05.309531 7788 consensus_queue.cc:237] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:05.312697 6508 catalog_manager.cc:5495] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d8b39082886f433c9b1a786b1c8e4328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } health_report { overall_health: UNKNOWN } } }
I20260620 08:55:05.336001 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.332663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.98.2:39981)
0620 08:55:05.332785 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.332790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.332803 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:05.332930 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.332933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.332991 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.333079 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:05.333328 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.333810 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.334639 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.334849 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.334940 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.335215 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:05.335627 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:05.335728 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.335805 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.335859 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:55:05.341449 7413 raft_consensus.cc:1261] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 1 FOLLOWER]: Refusing update from remote peer d8b39082886f433c9b1a786b1c8e4328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:05.341450 7701 raft_consensus.cc:1261] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Refusing update from remote peer d8b39082886f433c9b1a786b1c8e4328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:05.342082 7789 consensus_queue.cc:1035] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:05.342283 7788 consensus_queue.cc:1035] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:05.351578 7792 mvcc.cc:205] Tried to move back new op lower bound from 7298849609072054272 to 7298849608950284288. Current Snapshot: MvccSnapshot[applied={T|T < 7298849609072054272}]
I20260620 08:55:05.352113 7794 mvcc.cc:205] Tried to move back new op lower bound from 7298849609072054272 to 7298849608950284288. Current Snapshot: MvccSnapshot[applied={T|T < 7298849609072054272}]
I20260620 08:55:05.362223 6508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:05.362391 6508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:05.366354 6508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c3aed0df60b4a10ae10f6ab759f4ce7: Sending DeleteTablet for 3 replicas of tablet 52af58902da24c02baada9b5fcc88e6d
I20260620 08:55:05.367277 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 52af58902da24c02baada9b5fcc88e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49262
I20260620 08:55:05.367442 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 52af58902da24c02baada9b5fcc88e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36346
I20260620 08:55:05.368065 7802 tablet_replica.cc:330] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: stopping tablet replica
I20260620 08:55:05.368294 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet 52af58902da24c02baada9b5fcc88e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37096
I20260620 08:55:05.368395 7802 raft_consensus.cc:2229] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:05.368746 7804 tablet_replica.cc:330] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: stopping tablet replica
I20260620 08:55:05.368743 7802 raft_consensus.cc:2258] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:05.369084 7804 raft_consensus.cc:2229] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:05.369424 7804 raft_consensus.cc:2258] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:05.370334 7802 ts_tablet_manager.cc:1905] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:05.371182 7804 ts_tablet_manager.cc:1905] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:05.371717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6477
I20260620 08:55:05.374163 7802 ts_tablet_manager.cc:1918] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:05.374269 7802 log.cc:1196] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/52af58902da24c02baada9b5fcc88e6d
I20260620 08:55:05.374583 7802 ts_tablet_manager.cc:1939] T 52af58902da24c02baada9b5fcc88e6d P dbb7ef58462741c6bde02a0a9c023fa9: Deleting consensus metadata
I20260620 08:55:05.375083 7804 ts_tablet_manager.cc:1918] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:05.375388 7804 log.cc:1196] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/52af58902da24c02baada9b5fcc88e6d
I20260620 08:55:05.375761 7804 ts_tablet_manager.cc:1939] T 52af58902da24c02baada9b5fcc88e6d P eee75217f62c468dba0fcc893ec81db7: Deleting consensus metadata
W20260620 08:55:05.378626 7632 connection.cc:535] server connection from 127.0.0.1:37096 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:55:05.379529 7803 tablet_replica.cc:330] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: stopping tablet replica
I20260620 08:55:05.379850 7803 raft_consensus.cc:2229] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:05.380198 7803 raft_consensus.cc:2258] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:05.381484 7803 ts_tablet_manager.cc:1905] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:05.384137 7803 ts_tablet_manager.cc:1918] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:05.384268 7803 log.cc:1196] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/52af58902da24c02baada9b5fcc88e6d
I20260620 08:55:05.384577 7803 ts_tablet_manager.cc:1939] T 52af58902da24c02baada9b5fcc88e6d P d8b39082886f433c9b1a786b1c8e4328: Deleting consensus metadata
W20260620 08:55:05.385532 7496 connection.cc:441] server connection from 127.0.0.1:36346 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36346 (request call id 1) could send its response
W20260620 08:55:05.386029 7360 connection.cc:441] server connection from 127.0.0.1:49262 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49262 (request call id 1) could send its response
I20260620 08:55:05.391456 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.370748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.0.98.61:38155)
0620 08:55:05.370858 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.370862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.374609 (+ 3747us) server_negotiation.cc:407] Connection header received
0620 08:55:05.375353 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.375356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.375412 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.375501 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.383741 (+ 8240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.384572 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.385802 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.386158 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.386255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.388286 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.388290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.388292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.388319 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.389824 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.390245 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.390249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.390250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.390319 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.390563 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.390567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.390568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.390755 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.390891 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.391196 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.391309 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:55:05.396710 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.386509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.98.61:38155)
0620 08:55:05.386617 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.386621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.388398 (+ 1777us) server_negotiation.cc:407] Connection header received
0620 08:55:05.388563 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.388566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.388610 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.388658 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.389439 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.390202 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.391369 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.391708 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.391779 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.393384 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.393388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.393390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.393413 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.394901 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.395344 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.395347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.395349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.395428 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.395778 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.395782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.395783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.395982 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.396098 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.396449 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.396557 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:05.399760 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.370746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.0.98.60:38523)
0620 08:55:05.370867 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.370871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.391704 (+ 20833us) server_negotiation.cc:407] Connection header received
0620 08:55:05.391870 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.391873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.391926 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.391976 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.392833 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.393439 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.394534 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.394882 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.394955 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.396452 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.396456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.396459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.396480 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.398237 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.398621 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.398624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.398626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.398694 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.398960 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.398964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.398966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.399126 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.399233 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.399432 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.399582 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260620 08:55:05.401605 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:05.387537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.98.60:38523)
0620 08:55:05.387716 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:05.387720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:05.392400 (+ 4680us) server_negotiation.cc:407] Connection header received
0620 08:55:05.393491 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:05.393494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:05.393537 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:05.393583 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:05.394333 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.395105 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.396422 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:05.396743 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:05.396810 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:05.398522 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:05.398526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:05.398528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:05.398551 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:05.400152 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.400556 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.400560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.400561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.400639 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:05.400837 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:05.400840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:05.400842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:05.401032 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:05.401155 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:05.401358 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 08:55:05.401462 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 08:55:05.803265 7467 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.1:46045
I20260620 08:55:06.361996 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.361570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.1:45561)
0620 08:55:06.361828 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.361904 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":52,"threads_started":1}
I20260620 08:55:06.362306 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.361494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:47849)
0620 08:55:06.361927 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.362045 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":82,"threads_started":1}
W20260620 08:55:06.362426 7472 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
W20260620 08:55:06.362881 7609 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
I20260620 08:55:06.363024 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.362664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:47869)
0620 08:55:06.362860 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.362935 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
W20260620 08:55:06.363391 7747 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
I20260620 08:55:06.866287 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c3aed0df60b4a10ae10f6ab759f4ce7)
I20260620 08:55:06.866478 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } }
I20260620 08:55:06.867498 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c3aed0df60b4a10ae10f6ab759f4ce7)
I20260620 08:55:06.867532 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477), bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60:38523)
I20260620 08:55:06.867951 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } }
I20260620 08:55:06.868525 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.867868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34666)
0620 08:55:06.868296 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.868397 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":74,"threads_started":1}
W20260620 08:55:06.868743 7095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
I20260620 08:55:06.869769 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.869479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34670)
0620 08:55:06.869616 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.869684 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 08:55:06.869997 7817 leader_election.cc:288] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477), 7f27f25fbfe44a4d949940405fa61719 (127.0.98.61:38155)
I20260620 08:55:06.870438 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.869690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34686)
0620 08:55:06.870257 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.870336 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":64,"threads_started":1}
W20260620 08:55:06.870626 7279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
W20260620 08:55:06.871161 7095 leader_election.cc:334] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
I20260620 08:55:06.871783 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.871459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34696)
0620 08:55:06.871630 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.871702 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 08:55:06.872025 7279 leader_election.cc:334] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
I20260620 08:55:06.872691 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.868172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.98.60:38523)
0620 08:55:06.868413 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:06.868417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:06.868428 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:06.868471 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:06.868473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:06.868512 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:06.868582 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:06.868828 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.869491 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.871212 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.871868 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.871992 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.872069 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:06.872461 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 08:55:06.872577 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260620 08:55:06.872802 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.867834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38523 (local address 127.0.0.1:33598)
0620 08:55:06.868153 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.868172 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:06.868292 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:06.868615 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:06.868618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:06.868638 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:06.868731 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.868735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.869651 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.869655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.871013 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.871030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.872126 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.872130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.872314 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.872344 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 08:55:06.872430 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:55:06.872548 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":39,"thread_start_us":75,"threads_started":1}
I20260620 08:55:06.873425 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f27f25fbfe44a4d949940405fa61719" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" is_pre_election: true
I20260620 08:55:06.873849 7303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f27f25fbfe44a4d949940405fa61719 in term 1.
I20260620 08:55:06.874305 7096 leader_election.cc:302] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f27f25fbfe44a4d949940405fa61719, bc00eb79d59d4c4c9ab8e617b48f1304; no voters: 2c3aed0df60b4a10ae10f6ab759f4ce7
I20260620 08:55:06.874562 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 08:55:06.874619 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c3aed0df60b4a10ae10f6ab759f4ce7)
I20260620 08:55:06.874655 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:55:06.874763 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.870255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.98.61:38155)
0620 08:55:06.870811 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:06.870819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:06.870831 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:06.870864 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:06.870866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:06.870904 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:06.870969 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:06.871323 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.872229 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.873711 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.874254 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.874350 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.874414 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:06.874571 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 08:55:06.874638 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":491,"thread_start_us":87,"threads_started":1}
I20260620 08:55:06.874763 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.870225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38155 (local address 127.0.0.1:35438)
0620 08:55:06.870527 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.870541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:06.870634 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:06.871085 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:06.871088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:06.871107 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:06.871196 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.871202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.872348 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.872351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.873564 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.873575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.874361 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.874365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.874464 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.874489 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 08:55:06.874557 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:55:06.874638 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":83,"threads_started":1}
I20260620 08:55:06.875438 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f27f25fbfe44a4d949940405fa61719" is_pre_election: true
I20260620 08:55:06.875860 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } }
I20260620 08:55:06.876194 7119 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00eb79d59d4c4c9ab8e617b48f1304 in current term 2: Already voted for candidate 7f27f25fbfe44a4d949940405fa61719 in this term.
I20260620 08:55:06.876250 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [CANDIDATE]: Term 2 election: Requested vote from peers 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477), bc00eb79d59d4c4c9ab8e617b48f1304 (127.0.98.60:38523)
I20260620 08:55:06.876621 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.876375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34698)
0620 08:55:06.876509 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.876559 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 08:55:06.876600 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f27f25fbfe44a4d949940405fa61719" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304"
I20260620 08:55:06.876662 7280 leader_election.cc:302] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00eb79d59d4c4c9ab8e617b48f1304; no voters: 2c3aed0df60b4a10ae10f6ab759f4ce7, 7f27f25fbfe44a4d949940405fa61719
I20260620 08:55:06.876722 7303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:55:06.877250 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.877042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34710)
0620 08:55:06.877148 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.877197 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 08:55:06.877377 7095 leader_election.cc:334] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
I20260620 08:55:06.877758 7303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f27f25fbfe44a4d949940405fa61719 in term 2.
I20260620 08:55:06.877877 7817 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:55:06.878014 7096 leader_election.cc:302] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f27f25fbfe44a4d949940405fa61719, bc00eb79d59d4c4c9ab8e617b48f1304; no voters: 2c3aed0df60b4a10ae10f6ab759f4ce7
I20260620 08:55:06.878166 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:55:06.878376 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [term 2 LEADER]: Becoming Leader. State: Replica: 7f27f25fbfe44a4d949940405fa61719, State: Running, Role: LEADER
I20260620 08:55:06.878610 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } }
I20260620 08:55:06.879871 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f27f25fbfe44a4d949940405fa61719. Latest consensus state: current_term: 2 leader_uuid: "7f27f25fbfe44a4d949940405fa61719" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:06.879966 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:06.880236 7828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:06.882700 7828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:06.883208 7828 catalog_manager.cc:1259] Loaded cluster ID: a9e19b1412e840e9b9a52a6d69159b9a
I20260620 08:55:06.883272 7828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:06.883855 7828 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:06.884205 7828 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Loaded TSK: 0
I20260620 08:55:06.885061 7828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:55:06.908500 7108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:55:06.911465 7108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:06.914590 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.914282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34736)
0620 08:55:06.914449 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.914517 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 08:55:06.914757 7303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [term 2 FOLLOWER]: Refusing update from remote peer 7f27f25fbfe44a4d949940405fa61719: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 08:55:06.915480 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.915238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.0.1:34748)
0620 08:55:06.915364 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.915418 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 08:55:06.915455 7826 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 08:55:06.915642 7095 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 -> Peer 2c3aed0df60b4a10ae10f6ab759f4ce7 (127.0.98.62:36477): Couldn't send request to peer 2c3aed0df60b4a10ae10f6ab759f4ce7. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 08:55:06.917418 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f27f25fbfe44a4d949940405fa61719. Latest consensus state: current_term: 2 leader_uuid: "7f27f25fbfe44a4d949940405fa61719" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:06.917573 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:06.919034 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7f27f25fbfe44a4d949940405fa61719" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:06.919174 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:06.919459 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7f27f25fbfe44a4d949940405fa61719" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c3aed0df60b4a10ae10f6ab759f4ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36477 } } peers { permanent_uuid: "7f27f25fbfe44a4d949940405fa61719" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38155 } attrs { promote: false } } peers { permanent_uuid: "bc00eb79d59d4c4c9ab8e617b48f1304" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38523 } attrs { promote: false } } }
I20260620 08:55:06.919581 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc00eb79d59d4c4c9ab8e617b48f1304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:55:06.930667 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.926160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45053 (local address 127.0.0.1:37104)
0620 08:55:06.926381 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.926391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:06.926450 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:06.927406 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:06.927408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:06.927422 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:06.927486 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.927491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.928489 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.928493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.929537 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.929548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.930281 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.930285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.930370 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.930392 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:06.930460 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:55:06.930525 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":41}
I20260620 08:55:06.930670 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.926506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.0.98.3:45053)
0620 08:55:06.927149 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:06.927154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:06.927169 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:55:06.927218 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:06.927221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:06.927268 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:06.927399 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:06.927587 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.928339 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.929672 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.930165 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.930265 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.930330 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:06.930481 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:55:06.930550 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":62,"threads_started":1}
I20260620 08:55:06.930927 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.926034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39981 (local address 127.0.0.1:36370)
0620 08:55:06.926168 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.926184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:06.926260 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:06.927049 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:06.927052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:06.927068 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:06.927166 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.927171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.928208 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.928211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.929641 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.929651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.930534 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.930538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.930628 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.930657 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 08:55:06.930706 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:55:06.930782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":232}
I20260620 08:55:06.931380 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.926420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46045 (local address 127.0.0.1:49270)
0620 08:55:06.926678 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.926690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:06.926760 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:06.927292 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:06.927295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:06.927308 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:06.927385 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.927390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.928154 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.928157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.929759 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:06.929766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.930935 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.930939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:06.931102 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.931128 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 08:55:06.931178 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:55:06.931245 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":80,"thread_start_us":114,"threads_started":1}
I20260620 08:55:06.931272 7681 tablet_service.cc:1467] Processing CreateTablet for tablet 42902b65f58a40fcaf832f557da2d8ab (DEFAULT_TABLE table=test-table [id=8c0b4da79c0c48b6baa36b166fe8c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:06.931558 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.926122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.0.98.2:39981)
0620 08:55:06.926786 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:06.926791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:06.926809 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:55:06.926866 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:06.926869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:06.926923 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:06.927025 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:06.927269 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.928108 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.929778 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.930381 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.931207 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.931282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:06.931319 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 08:55:06.931407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":101,"threads_started":1}
I20260620 08:55:06.931861 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42902b65f58a40fcaf832f557da2d8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:06.932188 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.926822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.0.98.1:46045)
0620 08:55:06.927113 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:06.927118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:06.927129 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:06.927173 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:06.927176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:06.927220 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:06.927288 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:06.927490 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.928058 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.929875 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:06.930641 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:06.931832 (+ 1191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:06.931987 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:06.932022 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 08:55:06.932088 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":141,"threads_started":1}
I20260620 08:55:06.932758 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 42902b65f58a40fcaf832f557da2d8ab (DEFAULT_TABLE table=test-table [id=8c0b4da79c0c48b6baa36b166fe8c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:06.933084 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42902b65f58a40fcaf832f557da2d8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:06.934846 7839 tablet_bootstrap.cc:492] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Bootstrap starting.
I20260620 08:55:06.935268 7840 tablet_bootstrap.cc:492] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9: Bootstrap starting.
I20260620 08:55:06.935214 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 42902b65f58a40fcaf832f557da2d8ab (DEFAULT_TABLE table=test-table [id=8c0b4da79c0c48b6baa36b166fe8c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:06.935659 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42902b65f58a40fcaf832f557da2d8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:06.936149 7839 tablet_bootstrap.cc:654] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:06.936311 7840 tablet_bootstrap.cc:654] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:06.937637 7839 tablet_bootstrap.cc:492] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: No bootstrap required, opened a new log
I20260620 08:55:06.937685 7840 tablet_bootstrap.cc:492] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9: No bootstrap required, opened a new log
I20260620 08:55:06.937755 7839 ts_tablet_manager.cc:1397] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:55:06.937755 7840 ts_tablet_manager.cc:1397] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:55:06.938026 7843 tablet_bootstrap.cc:492] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Bootstrap starting.
I20260620 08:55:06.938215 7840 raft_consensus.cc:348] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:06.938234 7839 raft_consensus.cc:348] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:06.938346 7839 raft_consensus.cc:374] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:06.938346 7840 raft_consensus.cc:374] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:06.938392 7840 raft_consensus.cc:729] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb7ef58462741c6bde02a0a9c023fa9, State: Initialized, Role: FOLLOWER
I20260620 08:55:06.938390 7839 raft_consensus.cc:729] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee75217f62c468dba0fcc893ec81db7, State: Initialized, Role: FOLLOWER
I20260620 08:55:06.938517 7839 consensus_queue.cc:260] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:06.938517 7840 consensus_queue.cc:260] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:06.938874 7840 ts_tablet_manager.cc:1428] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:06.938886 7839 ts_tablet_manager.cc:1428] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260620 08:55:06.939507 7750 tablet.cc:2369] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:55:06.939821 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.939358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:49905)
0620 08:55:06.939656 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.939717 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260620 08:55:06.940219 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.939882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.1:47289)
0620 08:55:06.940082 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.940133 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260620 08:55:06.940542 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.940269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:50541)
0620 08:55:06.940423 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.940477 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 08:55:06.941303 7843 tablet_bootstrap.cc:654] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:06.942508 7843 tablet_bootstrap.cc:492] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: No bootstrap required, opened a new log
I20260620 08:55:06.942715 7843 ts_tablet_manager.cc:1397] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:55:06.943130 7474 heartbeater.cc:503] Master 127.0.98.61:38155 was elected leader, sending a full tablet report...
I20260620 08:55:06.943168 7843 raft_consensus.cc:348] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:06.943261 7843 raft_consensus.cc:374] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:06.943281 7843 raft_consensus.cc:729] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b39082886f433c9b1a786b1c8e4328, State: Initialized, Role: FOLLOWER
I20260620 08:55:06.943409 7843 consensus_queue.cc:260] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:06.943768 7843 ts_tablet_manager.cc:1428] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:06.944569 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.944069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:43459)
0620 08:55:06.944309 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.944379 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260620 08:55:06.945190 7611 heartbeater.cc:503] Master 127.0.98.61:38155 was elected leader, sending a full tablet report...
I20260620 08:55:06.945657 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:06.945438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:49935)
0620 08:55:06.945538 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:06.945591 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:55:06.990566 7612 tablet.cc:2369] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:55:07.056384 7845 raft_consensus.cc:482] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:07.056533 7845 raft_consensus.cc:504] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:07.057724 7845 leader_election.cc:288] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981), dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045)
I20260620 08:55:07.062584 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.058158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46045 (local address 127.0.98.3:35885)
0620 08:55:07.058442 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.058454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:07.058590 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:07.059079 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:07.059081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:07.059098 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:07.059174 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:07.059178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.060123 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.060127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:07.061321 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:07.061332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.062186 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.062190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:07.062279 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:07.062306 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 08:55:07.062366 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:55:07.062434 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260620 08:55:07.062645 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.058268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35885 (local address 127.0.98.1:46045)
0620 08:55:07.058374 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:07.058379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:07.058547 (+ 168us) server_negotiation.cc:407] Connection header received
0620 08:55:07.058841 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:07.058846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:07.058932 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:07.059060 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:07.059279 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.060007 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.061521 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.062047 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.062149 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:07.062339 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:07.062389 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 08:55:07.062498 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:55:07.062732 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.057925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39981 (local address 127.0.98.3:48779)
0620 08:55:07.058066 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.058083 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:07.058196 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:07.058482 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:07.058485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:07.058519 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:07.058669 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:07.058675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.059900 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.059904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:07.061321 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:07.061332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.062342 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.062346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:07.062435 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:07.062457 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:07.062545 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:55:07.062624 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":39,"spinlock_wait_cycles":66304}
I20260620 08:55:07.062794 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.058035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48779 (local address 127.0.98.2:39981)
0620 08:55:07.058174 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:07.058178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:07.058194 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:55:07.058325 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:07.058328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:07.058378 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:07.058451 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:07.058841 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.059748 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.061481 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.062186 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.062300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:07.062357 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:07.062585 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 08:55:07.062678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:55:07.063222 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42902b65f58a40fcaf832f557da2d8ab" candidate_uuid: "eee75217f62c468dba0fcc893ec81db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" is_pre_election: true
I20260620 08:55:07.063380 7413 raft_consensus.cc:2454] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee75217f62c468dba0fcc893ec81db7 in term 0.
I20260620 08:55:07.063397 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42902b65f58a40fcaf832f557da2d8ab" candidate_uuid: "eee75217f62c468dba0fcc893ec81db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b39082886f433c9b1a786b1c8e4328" is_pre_election: true
I20260620 08:55:07.063656 7563 raft_consensus.cc:2454] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee75217f62c468dba0fcc893ec81db7 in term 0.
I20260620 08:55:07.063782 7633 leader_election.cc:302] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb7ef58462741c6bde02a0a9c023fa9, eee75217f62c468dba0fcc893ec81db7; no voters:
I20260620 08:55:07.064035 7845 raft_consensus.cc:2781] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:07.064087 7845 raft_consensus.cc:482] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:07.064113 7845 raft_consensus.cc:3037] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:07.065194 7845 raft_consensus.cc:504] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:07.065510 7845 leader_election.cc:288] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [CANDIDATE]: Term 1 election: Requested vote from peers d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981), dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045)
I20260620 08:55:07.065832 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42902b65f58a40fcaf832f557da2d8ab" candidate_uuid: "eee75217f62c468dba0fcc893ec81db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b39082886f433c9b1a786b1c8e4328"
I20260620 08:55:07.065879 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42902b65f58a40fcaf832f557da2d8ab" candidate_uuid: "eee75217f62c468dba0fcc893ec81db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb7ef58462741c6bde02a0a9c023fa9"
I20260620 08:55:07.065950 7563 raft_consensus.cc:3037] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:07.065982 7413 raft_consensus.cc:3037] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:07.066869 7413 raft_consensus.cc:2454] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee75217f62c468dba0fcc893ec81db7 in term 1.
I20260620 08:55:07.067051 7563 raft_consensus.cc:2454] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee75217f62c468dba0fcc893ec81db7 in term 1.
I20260620 08:55:07.067132 7633 leader_election.cc:302] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb7ef58462741c6bde02a0a9c023fa9, eee75217f62c468dba0fcc893ec81db7; no voters:
I20260620 08:55:07.067322 7845 raft_consensus.cc:2781] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:07.067550 7845 raft_consensus.cc:686] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 1 LEADER]: Becoming Leader. State: Replica: eee75217f62c468dba0fcc893ec81db7, State: Running, Role: LEADER
I20260620 08:55:07.067809 7845 consensus_queue.cc:237] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } }
I20260620 08:55:07.067895 7749 heartbeater.cc:503] Master 127.0.98.61:38155 was elected leader, sending a full tablet report...
I20260620 08:55:07.068202 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.067995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:33279)
0620 08:55:07.068083 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.068137 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 08:55:07.070302 7108 catalog_manager.cc:5495] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eee75217f62c468dba0fcc893ec81db7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eee75217f62c468dba0fcc893ec81db7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eee75217f62c468dba0fcc893ec81db7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45053 } health_report { overall_health: HEALTHY } } }
I20260620 08:55:07.089890 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.085566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.0.98.3:45053)
0620 08:55:07.085670 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:07.085673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:07.085686 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:07.085781 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:07.085783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:07.085839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:07.085925 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:07.086093 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.086581 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.087900 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:07.088151 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:07.088271 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:07.088623 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:07.089272 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:07.089384 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:07.089590 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:55:07.089689 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:55:07.093747 7413 raft_consensus.cc:1261] T 42902b65f58a40fcaf832f557da2d8ab P dbb7ef58462741c6bde02a0a9c023fa9 [term 1 FOLLOWER]: Refusing update from remote peer eee75217f62c468dba0fcc893ec81db7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:07.094148 7563 raft_consensus.cc:1261] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Refusing update from remote peer eee75217f62c468dba0fcc893ec81db7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:07.094475 7852 consensus_queue.cc:1035] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb7ef58462741c6bde02a0a9c023fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:07.094753 7845 consensus_queue.cc:1035] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b39082886f433c9b1a786b1c8e4328" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:07.096951 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.096652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:35521)
0620 08:55:07.096804 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.096864 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 08:55:07.097421 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.096785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.1:46509)
0620 08:55:07.097295 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.097350 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444}
I20260620 08:55:07.100479 7856 mvcc.cc:205] Tried to move back new op lower bound from 7298849616248508416 to 7298849616152268800. Current Snapshot: MvccSnapshot[applied={T|T < 7298849616248508416}]
I20260620 08:55:07.101100 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.100890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.1:33705)
0620 08:55:07.100998 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.101043 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 08:55:07.102831 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.102450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.2:47017)
0620 08:55:07.102566 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.102747 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 08:55:07.104125 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.099876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:36699)
0620 08:55:07.103948 (+ 4072us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.104006 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4006}
I20260620 08:55:07.105201 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.104606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36477 (local address 127.0.98.3:32851)
0620 08:55:07.104999 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.105096 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260620 08:55:07.110340 7107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:07.110493 7107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:07.114509 7107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f27f25fbfe44a4d949940405fa61719: Sending DeleteTablet for 3 replicas of tablet 42902b65f58a40fcaf832f557da2d8ab
I20260620 08:55:07.115516 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 42902b65f58a40fcaf832f557da2d8ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49270
I20260620 08:55:07.115700 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 42902b65f58a40fcaf832f557da2d8ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36370
I20260620 08:55:07.115558 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet 42902b65f58a40fcaf832f557da2d8ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37104
I20260620 08:55:07.116043 7865 tablet_replica.cc:330] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: stopping tablet replica
I20260620 08:55:07.116191 7865 raft_consensus.cc:2229] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:07.116333 7865 raft_consensus.cc:2258] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:07.117029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 7253
I20260620 08:55:07.119148 7865 ts_tablet_manager.cc:1905] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:07.119436 7866 tablet_replica.cc:330] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: stopping tablet replica
I20260620 08:55:07.119562 7866 raft_consensus.cc:2229] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:07.119894 7866 raft_consensus.cc:2258] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:07.120563 7866 ts_tablet_manager.cc:1905] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:07.121445 7865 ts_tablet_manager.cc:1918] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:07.121512 7865 log.cc:1196] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/42902b65f58a40fcaf832f557da2d8ab
I20260620 08:55:07.122030 7865 ts_tablet_manager.cc:1939] T 42902b65f58a40fcaf832f557da2d8ab P d8b39082886f433c9b1a786b1c8e4328: Deleting consensus metadata
I20260620 08:55:07.123062 7094 catalog_manager.cc:4841] TS d8b39082886f433c9b1a786b1c8e4328 (127.0.98.2:39981): tablet 42902b65f58a40fcaf832f557da2d8ab (table test-table [id=8c0b4da79c0c48b6baa36b166fe8c2af]) successfully deleted
I20260620 08:55:07.123294 7866 ts_tablet_manager.cc:1918] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:07.123370 7866 log.cc:1196] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/42902b65f58a40fcaf832f557da2d8ab
I20260620 08:55:07.123663 7866 ts_tablet_manager.cc:1939] T 42902b65f58a40fcaf832f557da2d8ab P eee75217f62c468dba0fcc893ec81db7: Deleting consensus metadata
I20260620 08:55:07.124351 7095 catalog_manager.cc:4841] TS eee75217f62c468dba0fcc893ec81db7 (127.0.98.3:45053): tablet 42902b65f58a40fcaf832f557da2d8ab (table test-table [id=8c0b4da79c0c48b6baa36b166fe8c2af]) successfully deleted
I20260620 08:55:07.127810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 7480
I20260620 08:55:07.128623 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:07.127924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46045 (local address 127.0.0.1:49280)
0620 08:55:07.128491 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:07.128545 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
W20260620 08:55:07.128827 7094 catalog_manager.cc:4584] TS dbb7ef58462741c6bde02a0a9c023fa9 (127.0.98.1:46045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42902b65f58a40fcaf832f557da2d8ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46045: connect: Connection refused (error 111)
I20260620 08:55:07.136624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 7616
I20260620 08:55:07.141808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 6973
I20260620 08:55:07.146873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 7081
2026-06-20T08:55:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5140 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:07 dist-test-slave-fb1v krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:07 dist-test-slave-fb1v krb5kdc[7869](info): set up 2 sockets
Jun 20 08:55:07 dist-test-slave-fb1v krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260620 08:55:08.128600 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 08:55:08.249197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:08.360549 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:08.360814 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:08.360853 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:08.364856 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:08.364908 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:55:08.364925 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:08.364941 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:08.364962 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:08.364976 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:08.370128 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39373
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:08.371085 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:08.372071 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:08.379846 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:08.379858 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:08.379917 7882 server_base.cc:1034] running on GCE node
W20260620 08:55:08.379834 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:08.380568 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:08.381603 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:08.382792 7882 hybrid_clock.cc:648] HybridClock initialized: now 1781945708382768 us; error 38 us; skew 500 ppm
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:08.388096 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 08:55:08.388935 7882 webserver.cc:460] Webserver started at http://127.0.98.62:41319/ using document root <none> and password file <none>
I20260620 08:55:08.389462 7882 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:08.389518 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:08.389684 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:08.391665 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "be5727c5946443aea20a7726b5df57bb"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "3a200b89a2ad5504662ae7ba7cef39c8"
server_key_iv: "f5a61859f011508bcd98e198ca7a88f7"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.392177 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "be5727c5946443aea20a7726b5df57bb"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "3a200b89a2ad5504662ae7ba7cef39c8"
server_key_iv: "f5a61859f011508bcd98e198ca7a88f7"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.395592 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:08.398221 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:08.399587 7882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 08:55:08.399730 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "be5727c5946443aea20a7726b5df57bb"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "3a200b89a2ad5504662ae7ba7cef39c8"
server_key_iv: "f5a61859f011508bcd98e198ca7a88f7"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.399827 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:08.411387 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:08.414422 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:08.414578 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:08.421717 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39373
I20260620 08:55:08.421757 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39373 every 8 connection(s)
I20260620 08:55:08.422591 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:08.425318 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:08.425590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 7882
I20260620 08:55:08.425709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:08.425981 392 external_mini_cluster.cc:1428] Setting key 100a21a388877f2e4c00cd9056c513e2
I20260620 08:55:08.431387 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Bootstrap starting.
I20260620 08:55:08.433555 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Neither blocks nor log segments found. Creating new log.
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:08.434307 7960 log.cc:824] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:08.436076 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: No bootstrap required, opened a new log
I20260620 08:55:08.438899 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5727c5946443aea20a7726b5df57bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39373 } }
I20260620 08:55:08.439098 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.427283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.0.98.62:39373)
0620 08:55:08.427726 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:08.427740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:08.427774 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:55:08.428427 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:08.428451 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:08.428760 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:08.429055 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:08.429780 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.430872 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.431735 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.432191 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.432331 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.434634 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:08.434639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:08.434652 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:08.434679 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:08.436615 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.436992 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.436997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.437008 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.437106 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.437343 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.437346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.437348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.437724 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:08.437888 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:08.438154 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 08:55:08.438405 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":133,"threads_started":1}
I20260620 08:55:08.439118 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:08.439169 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5727c5946443aea20a7726b5df57bb, State: Initialized, Role: FOLLOWER
I20260620 08:55:08.439561 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5727c5946443aea20a7726b5df57bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39373 } }
I20260620 08:55:08.439714 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:08.439774 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:08.439873 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:08.440680 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5727c5946443aea20a7726b5df57bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39373 } }
I20260620 08:55:08.440991 7960 leader_election.cc:302] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5727c5946443aea20a7726b5df57bb; no voters:
I20260620 08:55:08.441278 7960 leader_election.cc:288] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:08.441402 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:08.441612 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [term 1 LEADER]: Becoming Leader. State: Replica: be5727c5946443aea20a7726b5df57bb, State: Running, Role: LEADER
I20260620 08:55:08.441892 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5727c5946443aea20a7726b5df57bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39373 } }
I20260620 08:55:08.442330 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:08.443320 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5727c5946443aea20a7726b5df57bb. Latest consensus state: current_term: 1 leader_uuid: "be5727c5946443aea20a7726b5df57bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5727c5946443aea20a7726b5df57bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39373 } } }
I20260620 08:55:08.443439 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:08.443715 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5727c5946443aea20a7726b5df57bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5727c5946443aea20a7726b5df57bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39373 } } }
I20260620 08:55:08.443807 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:08.443902 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:08.447089 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:08.453106 7974 catalog_manager.cc:1347] Generated new cluster ID: 6557ef4e67464dc88eca9f97b48ffffc
I20260620 08:55:08.453181 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:08.495570 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:08.496601 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:08.506449 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Generated new TSK 0
I20260620 08:55:08.507119 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 08:55:08.542114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39373
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:08.648897 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:08.649142 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:08.649204 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:08.653192 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:08.653254 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:08.653308 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:55:08.653364 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:08.658470 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:08.659463 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:08.660440 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:08.669720 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:08.669839 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:08.669732 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:08.670017 7986 server_base.cc:1034] running on GCE node
I20260620 08:55:08.670516 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:08.671249 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:08.672411 7986 hybrid_clock.cc:648] HybridClock initialized: now 1781945708672386 us; error 39 us; skew 500 ppm
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:08.674906 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 08:55:08.676025 7986 webserver.cc:460] Webserver started at http://127.0.98.1:38101/ using document root <none> and password file <none>
I20260620 08:55:08.676659 7986 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:08.676743 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:08.676926 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:08.678850 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "03daa5caaab64f3a935b90ec6f98b748"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "2be551712dceb7fc884a2717e5c5206b"
server_key_iv: "bdfc6135a9c3e6e997f100a1e4a0de89"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.679328 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03daa5caaab64f3a935b90ec6f98b748"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "2be551712dceb7fc884a2717e5c5206b"
server_key_iv: "bdfc6135a9c3e6e997f100a1e4a0de89"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.682842 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:55:08.685289 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:08.686510 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:08.686614 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "03daa5caaab64f3a935b90ec6f98b748"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "2be551712dceb7fc884a2717e5c5206b"
server_key_iv: "bdfc6135a9c3e6e997f100a1e4a0de89"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.686715 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:08.701784 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:08.705020 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:08.705206 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:08.706782 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:08.706835 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:08.706879 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:08.706941 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:08.716529 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36223
I20260620 08:55:08.716593 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36223 every 8 connection(s)
I20260620 08:55:08.717563 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:08.718441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 7986
I20260620 08:55:08.718561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:08.718904 392 external_mini_cluster.cc:1428] Setting key 01cf7b5b07e49dd6a2600d3dcfef0a41
I20260620 08:55:08.719965 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.707419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59047 (local address 127.0.98.62:39373)
0620 08:55:08.707561 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:08.707566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:08.708431 (+ 865us) server_negotiation.cc:407] Connection header received
0620 08:55:08.709508 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:08.709513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:08.709571 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:08.709680 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:08.711092 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.711953 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.713250 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.713608 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.713711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.716034 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:08.716038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:08.716041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:08.716070 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:08.717687 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.718156 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.718159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.718161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.718246 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.718698 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.718701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.718703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.718931 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:08.719031 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:08.719692 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 08:55:08.719819 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:55:08.721499 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.707829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.98.1:59047)
0620 08:55:08.708276 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:08.708325 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:08.709245 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:08.709860 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:08.709870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:08.710300 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:08.710936 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.710956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.712082 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.712086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.713149 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.713158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.713742 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.713745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.713876 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.714454 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:08.714469 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:08.715871 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:08.717789 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.717793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.717804 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.718047 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.718356 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.718359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.718361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.718532 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.719049 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:08.719055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:08.719338 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 08:55:08.719696 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 08:55:08.720548 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260620 08:55:08.722608 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39373
I20260620 08:55:08.722764 8117 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:08.723272 8117 heartbeater.cc:511] Master 127.0.98.62:39373 requested a full tablet report, sending...
I20260620 08:55:08.724821 7912 ts_manager.cc:194] Registered new tserver with Master: 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223)
I20260620 08:55:08.725893 7912 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59047
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 08:55:08.734174 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.726651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52965 (local address 127.0.98.62:39373)
0620 08:55:08.727766 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:08.727770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:08.727782 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:08.727812 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:08.727814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:08.727847 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:08.727921 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:08.728365 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.728855 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.729673 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.729913 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.729999 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.731124 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:08.731134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:08.731137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:08.731161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:08.732709 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.733116 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.733119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.733121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.733192 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.733425 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.733428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.733430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.733594 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:08.733675 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:08.733921 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 08:55:08.734016 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042}
I20260620 08:55:08.734182 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.726597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.98.1:52965)
0620 08:55:08.726844 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:08.726857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:08.726938 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:08.727958 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:08.727961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:08.728132 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:08.728223 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.728230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.728977 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.728980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.729569 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.729577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.730058 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.730061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.730134 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.730581 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:08.730594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:08.731001 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:08.732826 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.732829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.732831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.733010 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.733273 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.733276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.733279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.733350 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.733679 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:08.733682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:08.733778 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 08:55:08.733865 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 08:55:08.734001 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 08:55:08.747723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39373
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:08.853500 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:08.853770 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:08.853814 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:08.857841 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:08.857893 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:08.857931 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:55:08.857963 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:08.863078 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:08.864099 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:08.865025 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:08.874382 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:08.874411 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:08.874471 8126 server_base.cc:1034] running on GCE node
W20260620 08:55:08.874387 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:08.874982 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:08.875569 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:08.876771 8126 hybrid_clock.cc:648] HybridClock initialized: now 1781945708876754 us; error 28 us; skew 500 ppm
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:08.879298 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 08:55:08.880136 8126 webserver.cc:460] Webserver started at http://127.0.98.2:37605/ using document root <none> and password file <none>
I20260620 08:55:08.880645 8126 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:08.880723 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:08.880905 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:08.882861 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "767930edba384d7eb4c849c213b1d3b0"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "31304af0a343a4f44d312d93197a42c2"
server_key_iv: "9e45083f3f523123f6d3ad2aa1d0c3b3"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.883355 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "767930edba384d7eb4c849c213b1d3b0"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "31304af0a343a4f44d312d93197a42c2"
server_key_iv: "9e45083f3f523123f6d3ad2aa1d0c3b3"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.886793 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260620 08:55:08.889110 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:08.890419 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:08.890524 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "767930edba384d7eb4c849c213b1d3b0"
format_stamp: "Formatted at 2026-06-20 08:55:08 on dist-test-slave-fb1v"
server_key: "31304af0a343a4f44d312d93197a42c2"
server_key_iv: "9e45083f3f523123f6d3ad2aa1d0c3b3"
server_key_version: "encryptionkey@0"
I20260620 08:55:08.890635 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:08.902804 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:08.905745 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:08.905921 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:08.907298 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:08.907348 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:08.907401 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:08.907429 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:08.916673 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40695
Jun 20 08:55:08 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:08.917430 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40695 every 8 connection(s)
I20260620 08:55:08.917814 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:08.921538 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.908070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41723 (local address 127.0.98.62:39373)
0620 08:55:08.908203 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:08.908208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:08.909056 (+ 848us) server_negotiation.cc:407] Connection header received
0620 08:55:08.910074 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:08.910077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:08.910141 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:08.910215 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:08.911812 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.912671 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.914232 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.914557 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.914669 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.917790 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:08.917794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:08.917796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:08.917821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:08.919538 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.920135 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.920139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.920140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.920239 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.920560 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.920563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.920564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.920772 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:08.920861 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:08.921310 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 08:55:08.921421 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:55:08.922307 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.908439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.98.2:41723)
0620 08:55:08.908894 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:08.908942 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:08.909862 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:08.910525 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:08.910535 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:08.910927 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:08.911587 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.911646 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.912804 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.912807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.914095 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.914105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.914851 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.914856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.915083 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.915810 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:08.915838 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:08.917589 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:08.919760 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.919765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.919776 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.920011 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.920323 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.920325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.920326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.920449 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.920879 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:08.920885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:08.921110 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 08:55:08.921267 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 08:55:08.921505 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260620 08:55:08.923592 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39373
I20260620 08:55:08.923645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 8126
I20260620 08:55:08.923758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:55:08.923753 8257 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:08.924031 392 external_mini_cluster.cc:1428] Setting key 1b1a60da89698ede671b07b9335068e8
I20260620 08:55:08.924281 8257 heartbeater.cc:511] Master 127.0.98.62:39373 requested a full tablet report, sending...
I20260620 08:55:08.925269 7914 ts_manager.cc:194] Registered new tserver with Master: 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695)
I20260620 08:55:08.925942 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41723
I20260620 08:55:08.933329 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.925783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.98.2:44239)
0620 08:55:08.926109 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:08.926124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:08.926220 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:08.926442 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:08.926445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:08.926632 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:08.926759 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.926766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.927866 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.927870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.928459 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:08.928467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.928999 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.929002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:08.929077 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.929659 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:08.929670 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:08.929993 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:08.931823 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.931826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.931827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.932018 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.932314 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:08.932316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:08.932318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:08.932420 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:08.932888 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:08.932890 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:08.932969 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 08:55:08.933058 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:55:08.933173 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:55:08.933399 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:08.925854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44239 (local address 127.0.98.62:39373)
0620 08:55:08.925948 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:08.925952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:08.926184 (+ 232us) server_negotiation.cc:407] Connection header received
0620 08:55:08.926299 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:08.926302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:08.926346 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:08.926423 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:08.926872 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.927629 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.928570 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:08.928870 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:08.928982 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:08.930129 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:08.930133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:08.930136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:08.930162 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:08.931703 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.932111 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.932115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.932116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.932200 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:08.932512 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:08.932515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:08.932516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:08.932794 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:08.932902 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:08.933130 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 08:55:08.933243 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 08:55:08.950340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39373
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:09.063719 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:09.063987 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:09.064051 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:09.068063 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:09.068127 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:09.068183 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:55:09.068240 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:09.073976 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:09.075002 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:09.076040 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:09.085438 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:09.085528 8266 server_base.cc:1034] running on GCE node
W20260620 08:55:09.085484 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:09.085446 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:09.086146 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:09.086767 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:09.087970 8266 hybrid_clock.cc:648] HybridClock initialized: now 1781945709087952 us; error 34 us; skew 500 ppm
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:09.090787 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 08:55:09.091713 8266 webserver.cc:460] Webserver started at http://127.0.98.3:40723/ using document root <none> and password file <none>
I20260620 08:55:09.092214 8266 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:09.092269 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:09.092423 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:09.094360 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "89f05142c3754c4a98d79a771830a80c"
format_stamp: "Formatted at 2026-06-20 08:55:09 on dist-test-slave-fb1v"
server_key: "88b5f8991a05c77d1f0f13b1144f34b3"
server_key_iv: "c4f3846b8e914ea24ad19e333c10f9f7"
server_key_version: "encryptionkey@0"
I20260620 08:55:09.094837 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "89f05142c3754c4a98d79a771830a80c"
format_stamp: "Formatted at 2026-06-20 08:55:09 on dist-test-slave-fb1v"
server_key: "88b5f8991a05c77d1f0f13b1144f34b3"
server_key_iv: "c4f3846b8e914ea24ad19e333c10f9f7"
server_key_version: "encryptionkey@0"
I20260620 08:55:09.098335 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 08:55:09.100730 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:09.102051 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:09.102160 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "89f05142c3754c4a98d79a771830a80c"
format_stamp: "Formatted at 2026-06-20 08:55:09 on dist-test-slave-fb1v"
server_key: "88b5f8991a05c77d1f0f13b1144f34b3"
server_key_iv: "c4f3846b8e914ea24ad19e333c10f9f7"
server_key_version: "encryptionkey@0"
I20260620 08:55:09.102252 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:09.113586 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:09.116346 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:09.116504 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:09.117836 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:09.117884 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:09.117933 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:09.117959 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:09.127759 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38213
I20260620 08:55:09.128878 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:09.129669 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38213 every 8 connection(s)
I20260620 08:55:09.130828 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.118588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45111 (local address 127.0.98.62:39373)
0620 08:55:09.118708 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.118712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.119654 (+ 942us) server_negotiation.cc:407] Connection header received
0620 08:55:09.120634 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.120638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.120691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.120748 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.122058 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.122937 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.124391 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.124659 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.124752 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.127169 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:09.127172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:09.127175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:09.127203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:09.128941 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.129417 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.129420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.129422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.129506 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.129837 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.129840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.129842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.130023 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:09.130105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.130550 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.130669 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:55:09.131517 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.119026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.98.3:45111)
0620 08:55:09.119504 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.119542 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.120397 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.121122 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.121128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.121446 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:09.121926 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.121940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.123048 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.123051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.124249 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.124258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.124807 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.124817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.124985 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.125514 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:09.125529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:09.127017 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:09.129047 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:09.129053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:09.129066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:09.129300 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:09.129602 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:09.129605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:09.129607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:09.129757 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:09.130114 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:09.130119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:09.130334 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.130489 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.130776 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260620 08:55:09.133010 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39373
I20260620 08:55:09.133174 8397 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:09.133669 8397 heartbeater.cc:511] Master 127.0.98.62:39373 requested a full tablet report, sending...
I20260620 08:55:09.134699 7912 ts_manager.cc:194] Registered new tserver with Master: 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.135335 7912 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45111
I20260620 08:55:09.136744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 8266
I20260620 08:55:09.136868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:09.137156 392 external_mini_cluster.cc:1428] Setting key a29fd2b3302fed573525399b3e651e99
I20260620 08:55:09.139190 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:55:09.141842 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.134820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.98.3:35239)
0620 08:55:09.135140 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.135151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.135239 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.135441 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.135444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.135694 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:09.135795 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.135801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.136535 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.136538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.137340 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.137350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.137798 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.137802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.137892 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.138445 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:09.138459 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:09.138784 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:09.140364 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:09.140367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:09.140369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:09.140603 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:09.140870 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:09.140873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:09.140875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:09.140977 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:09.141437 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:09.141439 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:09.141515 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.141591 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.141707 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":87,"threads_started":1}
I20260620 08:55:09.141918 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.134891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35239 (local address 127.0.98.62:39373)
0620 08:55:09.135014 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.135018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.135219 (+ 201us) server_negotiation.cc:407] Connection header received
0620 08:55:09.135320 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.135322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.135359 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.135417 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.135894 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.136400 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.137450 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.137691 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.137799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.138909 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:09.138912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:09.138916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:09.138939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:09.140259 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.140685 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.140688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.140689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.140752 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.141112 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.141116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.141118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.141338 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:09.141434 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.141650 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.141776 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945709, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:55:09.163582 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:09.157483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.98.62:39373)
0620 08:55:09.157604 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.157607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.157618 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:09.157657 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.157659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.157695 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.157755 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.158498 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.159020 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.159952 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.160184 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.160270 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.163457 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39234: BlockingRecv error: recv got EOF from 127.0.0.1:39234 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945709, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:09.172375 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.165029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.0.98.62:39373)
0620 08:55:09.165219 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.165222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.165232 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:55:09.165268 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.165270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.165308 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.165369 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.165923 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.166416 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.167153 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.167360 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.167442 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.169684 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:09.169687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:09.169690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:09.169711 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:09.170948 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.171320 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.171323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.171325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.171395 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.171646 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.171650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.171652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.171815 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:09.171931 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.172129 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.172226 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 08:55:09.174955 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:55:09.178171 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:09.192533 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.186825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.0.98.3:38213)
0620 08:55:09.187107 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.187120 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.187454 (+ 334us) server_negotiation.cc:407] Connection header received
0620 08:55:09.187521 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.187525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.187693 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.187809 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.188258 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.188982 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.190667 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.191528 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.191674 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.191822 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.192263 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.192363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260620 08:55:09.192565 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.186247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.0.1:34462)
0620 08:55:09.187097 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.187111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.187201 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.187944 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.187947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.187971 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.188122 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.188129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.189426 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.189429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.190556 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.190564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.191786 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.191797 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.191987 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.192065 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.192166 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.192393 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":54,"spinlock_wait_cycles":1792,"thread_start_us":92,"threads_started":1}
I20260620 08:55:09.192637 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.186247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.0.98.2:40695)
0620 08:55:09.187310 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.187321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.187343 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:55:09.187414 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.187420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.187651 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.187804 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.188072 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.189275 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.190734 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.191905 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.192085 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.192351 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.192419 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.192514 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":93,"threads_started":1}
I20260620 08:55:09.192576 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.186113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40695 (local address 127.0.0.1:47158)
0620 08:55:09.186844 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.186892 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.187017 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.187801 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.187808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.187829 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.187957 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.187964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.189384 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.189388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.190614 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.190635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.192106 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.192110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.192193 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.192332 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.192390 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.192459 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":145,"thread_start_us":72,"threads_started":1}
I20260620 08:55:09.192924 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.186448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38213 (local address 127.0.0.1:53022)
0620 08:55:09.187281 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.187293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.187374 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.187944 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.187947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.187977 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.188131 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.188136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.189090 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.189093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.190543 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.190553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.191904 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.191908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.191991 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.192046 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.192149 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.192757 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":499,"spinlock_wait_cycles":7424,"thread_start_us":89,"threads_started":1}
I20260620 08:55:09.193954 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.186840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.98.1:36223)
0620 08:55:09.187417 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.187432 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.187453 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:55:09.187537 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.187542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.187755 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.187893 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.188243 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.189314 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.190697 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.191533 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.191663 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.191822 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.193729 (+ 1907us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.193821 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":95,"threads_started":1}
I20260620 08:55:09.195906 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 6a20be84ec364160954c6651bb24ac81 (DEFAULT_TABLE table=test-table [id=50e1ea6cf23c46aebd75c47918084b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.195904 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 6a20be84ec364160954c6651bb24ac81 (DEFAULT_TABLE table=test-table [id=50e1ea6cf23c46aebd75c47918084b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.196102 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 6a20be84ec364160954c6651bb24ac81 (DEFAULT_TABLE table=test-table [id=50e1ea6cf23c46aebd75c47918084b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.196980 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a20be84ec364160954c6651bb24ac81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.196985 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a20be84ec364160954c6651bb24ac81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.196981 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a20be84ec364160954c6651bb24ac81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.203411 8425 tablet_bootstrap.cc:492] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Bootstrap starting.
I20260620 08:55:09.204723 8426 tablet_bootstrap.cc:492] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Bootstrap starting.
I20260620 08:55:09.204723 8427 tablet_bootstrap.cc:492] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Bootstrap starting.
I20260620 08:55:09.206435 8425 tablet_bootstrap.cc:654] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.206768 8426 tablet_bootstrap.cc:654] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.206766 8427 tablet_bootstrap.cc:654] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.207211 8425 log.cc:824] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:09.207556 8427 log.cc:824] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:09.207554 8426 log.cc:824] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:09.209131 8427 tablet_bootstrap.cc:492] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: No bootstrap required, opened a new log
I20260620 08:55:09.209115 8425 tablet_bootstrap.cc:492] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: No bootstrap required, opened a new log
I20260620 08:55:09.209260 8426 tablet_bootstrap.cc:492] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: No bootstrap required, opened a new log
I20260620 08:55:09.209316 8425 ts_tablet_manager.cc:1397] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 08:55:09.209316 8427 ts_tablet_manager.cc:1397] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:55:09.209407 8426 ts_tablet_manager.cc:1397] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:55:09.212471 8427 raft_consensus.cc:348] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.212702 8427 raft_consensus.cc:374] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.212766 8427 raft_consensus.cc:729] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.213151 8427 consensus_queue.cc:260] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.213080 8425 raft_consensus.cc:348] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.213080 8426 raft_consensus.cc:348] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.213320 8426 raft_consensus.cc:374] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.213320 8425 raft_consensus.cc:374] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.213368 8425 raft_consensus.cc:729] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03daa5caaab64f3a935b90ec6f98b748, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.213368 8426 raft_consensus.cc:729] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.213780 8425 consensus_queue.cc:260] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.213780 8426 consensus_queue.cc:260] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.214040 8427 ts_tablet_manager.cc:1428] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:55:09.214391 8397 heartbeater.cc:503] Master 127.0.98.62:39373 was elected leader, sending a full tablet report...
I20260620 08:55:09.214591 8257 heartbeater.cc:503] Master 127.0.98.62:39373 was elected leader, sending a full tablet report...
I20260620 08:55:09.214684 8426 ts_tablet_manager.cc:1428] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 08:55:09.215395 8117 heartbeater.cc:503] Master 127.0.98.62:39373 was elected leader, sending a full tablet report...
I20260620 08:55:09.216382 8425 ts_tablet_manager.cc:1428] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 08:55:09.219789 8118 tablet.cc:2369] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:55:09.225939 8433 raft_consensus.cc:482] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:09.226119 8433 raft_consensus.cc:504] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.227217 8433 leader_election.cc:288] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695), 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.230913 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.227526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44105 (local address 127.0.98.3:38213)
0620 08:55:09.227684 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.227688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.227876 (+ 188us) server_negotiation.cc:407] Connection header received
0620 08:55:09.227972 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.227974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.228015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.228085 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.228315 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.228825 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.229934 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.230329 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.230419 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.230485 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.230720 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.230787 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:55:09.230919 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.227513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48887 (local address 127.0.98.2:40695)
0620 08:55:09.227665 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.227669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.227807 (+ 138us) server_negotiation.cc:407] Connection header received
0620 08:55:09.227919 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.227923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.227970 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.228027 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.228299 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.228812 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.229919 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.230322 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.230414 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.230477 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.230720 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.230795 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:55:09.230937 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.227406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40695 (local address 127.0.98.1:48887)
0620 08:55:09.227720 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.227733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.227827 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.228051 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.228053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.228069 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.228194 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.228200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.228911 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.228915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.229803 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.229812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.230428 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.230431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.230603 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.230628 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.230693 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.230786 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":74,"thread_start_us":76,"threads_started":1}
I20260620 08:55:09.230952 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.227447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38213 (local address 127.0.98.1:44105)
0620 08:55:09.227781 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.227794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.227895 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.228106 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.228109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.228124 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.228217 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.228222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.228929 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.228933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.229812 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.229821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.230438 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.230441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.230599 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.230626 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.230691 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.230802 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":110,"thread_start_us":109,"threads_started":1}
I20260620 08:55:09.231521 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a20be84ec364160954c6651bb24ac81" candidate_uuid: "03daa5caaab64f3a935b90ec6f98b748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f05142c3754c4a98d79a771830a80c" is_pre_election: true
I20260620 08:55:09.231520 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a20be84ec364160954c6651bb24ac81" candidate_uuid: "03daa5caaab64f3a935b90ec6f98b748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767930edba384d7eb4c849c213b1d3b0" is_pre_election: true
I20260620 08:55:09.231833 8211 raft_consensus.cc:2454] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03daa5caaab64f3a935b90ec6f98b748 in term 0.
I20260620 08:55:09.231834 8349 raft_consensus.cc:2454] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03daa5caaab64f3a935b90ec6f98b748 in term 0.
I20260620 08:55:09.232275 8003 leader_election.cc:302] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03daa5caaab64f3a935b90ec6f98b748, 767930edba384d7eb4c849c213b1d3b0; no voters:
I20260620 08:55:09.232523 8433 raft_consensus.cc:2781] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:09.232599 8433 raft_consensus.cc:482] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:09.232658 8433 raft_consensus.cc:3037] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.234030 8433 raft_consensus.cc:504] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.234375 8433 leader_election.cc:288] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [CANDIDATE]: Term 1 election: Requested vote from peers 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695), 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.234717 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a20be84ec364160954c6651bb24ac81" candidate_uuid: "03daa5caaab64f3a935b90ec6f98b748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f05142c3754c4a98d79a771830a80c"
I20260620 08:55:09.234719 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a20be84ec364160954c6651bb24ac81" candidate_uuid: "03daa5caaab64f3a935b90ec6f98b748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767930edba384d7eb4c849c213b1d3b0"
I20260620 08:55:09.234823 8349 raft_consensus.cc:3037] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.234828 8211 raft_consensus.cc:3037] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.235867 8349 raft_consensus.cc:2454] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03daa5caaab64f3a935b90ec6f98b748 in term 1.
I20260620 08:55:09.235975 8211 raft_consensus.cc:2454] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03daa5caaab64f3a935b90ec6f98b748 in term 1.
I20260620 08:55:09.236192 8002 leader_election.cc:302] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03daa5caaab64f3a935b90ec6f98b748, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:09.236363 8433 raft_consensus.cc:2781] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:09.236536 8433 raft_consensus.cc:686] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 LEADER]: Becoming Leader. State: Replica: 03daa5caaab64f3a935b90ec6f98b748, State: Running, Role: LEADER
I20260620 08:55:09.236871 8433 consensus_queue.cc:237] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.239759 7914 catalog_manager.cc:5495] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "03daa5caaab64f3a935b90ec6f98b748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } health_report { overall_health: UNKNOWN } } }
I20260620 08:55:09.257735 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.254319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.98.1:36223)
0620 08:55:09.254446 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.254451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.254465 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:55:09.254544 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.254546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.254601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.254701 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:09.254938 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.255437 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.256291 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.256502 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.256591 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.256928 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.257356 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.257460 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.257529 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.257587 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:55:09.263873 8349 raft_consensus.cc:1261] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Refusing update from remote peer 03daa5caaab64f3a935b90ec6f98b748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:09.263945 8211 raft_consensus.cc:1261] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 03daa5caaab64f3a935b90ec6f98b748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:09.264623 8436 consensus_queue.cc:1035] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:09.264798 8433 consensus_queue.cc:1035] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:09.274816 8440 mvcc.cc:205] Tried to move back new op lower bound from 7298849625135722496 to 7298849625036861440. Current Snapshot: MvccSnapshot[applied={T|T < 7298849625135722496}]
I20260620 08:55:09.279448 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:09.279656 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:09.282550 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Sending DeleteTablet for 3 replicas of tablet 6a20be84ec364160954c6651bb24ac81
I20260620 08:55:09.283311 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a20be84ec364160954c6651bb24ac81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260620 08:55:09.283325 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a20be84ec364160954c6651bb24ac81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34462
I20260620 08:55:09.283818 8451 tablet_replica.cc:330] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: stopping tablet replica
I20260620 08:55:09.283824 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a20be84ec364160954c6651bb24ac81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53022
I20260620 08:55:09.284152 8451 raft_consensus.cc:2229] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:09.284291 8452 tablet_replica.cc:330] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: stopping tablet replica
I20260620 08:55:09.284483 8451 raft_consensus.cc:2258] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:09.284669 8452 raft_consensus.cc:2229] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:09.285013 8452 raft_consensus.cc:2258] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:09.285301 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:55:09.285770 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:09.286283 8451 ts_tablet_manager.cc:1905] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:09.286724 8452 ts_tablet_manager.cc:1905] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:09.288990 8451 ts_tablet_manager.cc:1918] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:09.289090 8451 log.cc:1196] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/6a20be84ec364160954c6651bb24ac81
I20260620 08:55:09.289381 8451 ts_tablet_manager.cc:1939] T 6a20be84ec364160954c6651bb24ac81 P 767930edba384d7eb4c849c213b1d3b0: Deleting consensus metadata
I20260620 08:55:09.289580 8452 ts_tablet_manager.cc:1918] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:09.289719 8452 log.cc:1196] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/6a20be84ec364160954c6651bb24ac81
I20260620 08:55:09.289769 8450 tablet_replica.cc:330] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: stopping tablet replica
I20260620 08:55:09.290089 8450 raft_consensus.cc:2229] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:09.290150 8452 ts_tablet_manager.cc:1939] T 6a20be84ec364160954c6651bb24ac81 P 89f05142c3754c4a98d79a771830a80c: Deleting consensus metadata
I20260620 08:55:09.290462 8450 raft_consensus.cc:2258] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:09.291586 8191 tablet_service.cc:1467] Processing CreateTablet for tablet f6805c9a1fc44df9835a819307bb3bcb (DEFAULT_TABLE table=test-table [id=979047c1421640959ecdc922f55c6ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.291648 7898 catalog_manager.cc:4841] TS 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213): tablet 6a20be84ec364160954c6651bb24ac81 (table test-table [id=50e1ea6cf23c46aebd75c47918084b17]) successfully deleted
I20260620 08:55:09.291720 8331 tablet_service.cc:1467] Processing CreateTablet for tablet f6805c9a1fc44df9835a819307bb3bcb (DEFAULT_TABLE table=test-table [id=979047c1421640959ecdc922f55c6ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.291874 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6805c9a1fc44df9835a819307bb3bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.291949 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6805c9a1fc44df9835a819307bb3bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.292233 8051 tablet_service.cc:1467] Processing CreateTablet for tablet f6805c9a1fc44df9835a819307bb3bcb (DEFAULT_TABLE table=test-table [id=979047c1421640959ecdc922f55c6ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.292452 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6805c9a1fc44df9835a819307bb3bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.294024 8427 tablet_bootstrap.cc:492] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Bootstrap starting.
I20260620 08:55:09.294713 8425 tablet_bootstrap.cc:492] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Bootstrap starting.
I20260620 08:55:09.294965 8427 tablet_bootstrap.cc:654] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.295686 8426 tablet_bootstrap.cc:492] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Bootstrap starting.
I20260620 08:55:09.295861 8425 tablet_bootstrap.cc:654] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.296778 8426 tablet_bootstrap.cc:654] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.297065 8425 tablet_bootstrap.cc:492] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: No bootstrap required, opened a new log
I20260620 08:55:09.297147 8425 ts_tablet_manager.cc:1397] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 08:55:09.297590 8425 raft_consensus.cc:348] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.297662 8427 tablet_bootstrap.cc:492] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: No bootstrap required, opened a new log
I20260620 08:55:09.297703 8425 raft_consensus.cc:374] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.297734 8425 raft_consensus.cc:729] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03daa5caaab64f3a935b90ec6f98b748, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.297739 8427 ts_tablet_manager.cc:1397] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 08:55:09.297844 8425 consensus_queue.cc:260] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.298091 8425 ts_tablet_manager.cc:1428] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:55:09.298127 8427 raft_consensus.cc:348] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.298236 8427 raft_consensus.cc:374] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.298264 8427 raft_consensus.cc:729] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.298375 8427 consensus_queue.cc:260] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.298609 8427 ts_tablet_manager.cc:1428] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:09.299496 8426 tablet_bootstrap.cc:492] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: No bootstrap required, opened a new log
I20260620 08:55:09.299577 8426 ts_tablet_manager.cc:1397] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 08:55:09.300026 8426 raft_consensus.cc:348] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.300132 8426 raft_consensus.cc:374] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.300161 8426 raft_consensus.cc:729] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.300278 8426 consensus_queue.cc:260] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.300534 8426 ts_tablet_manager.cc:1428] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:55:09.302129 8450 ts_tablet_manager.cc:1905] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:09.304831 8450 ts_tablet_manager.cc:1918] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:09.304914 8450 log.cc:1196] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/6a20be84ec364160954c6651bb24ac81
I20260620 08:55:09.305171 8450 ts_tablet_manager.cc:1939] T 6a20be84ec364160954c6651bb24ac81 P 03daa5caaab64f3a935b90ec6f98b748: Deleting consensus metadata
I20260620 08:55:09.306376 7901 catalog_manager.cc:4841] TS 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223): tablet 6a20be84ec364160954c6651bb24ac81 (table test-table [id=50e1ea6cf23c46aebd75c47918084b17]) successfully deleted
I20260620 08:55:09.306679 7899 catalog_manager.cc:4841] TS 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695): tablet 6a20be84ec364160954c6651bb24ac81 (table test-table [id=50e1ea6cf23c46aebd75c47918084b17]) successfully deleted
W20260620 08:55:09.379956 8398 tablet.cc:2369] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:55:09.419056 8258 tablet.cc:2369] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:55:09.436028 8432 raft_consensus.cc:482] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:09.436221 8432 raft_consensus.cc:504] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.437294 8432 leader_election.cc:288] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.442284 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.437525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38213 (local address 127.0.98.2:55951)
0620 08:55:09.437903 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.437918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.438037 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.438427 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.438430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.438445 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.438557 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.438564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.439582 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.439586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.441124 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.441132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.441796 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.441800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.441905 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.441968 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.442040 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.442117 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":59,"thread_start_us":69,"threads_started":1}
I20260620 08:55:09.442320 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.437869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55951 (local address 127.0.98.3:38213)
0620 08:55:09.438015 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.438020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.438038 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:55:09.438173 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.438177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.438229 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.438276 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.438670 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.439446 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.441292 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.441678 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.441773 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.441837 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.442076 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.442168 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 08:55:09.442360 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.437875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41711 (local address 127.0.98.1:36223)
0620 08:55:09.437982 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.437986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.438231 (+ 245us) server_negotiation.cc:407] Connection header received
0620 08:55:09.438399 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.438402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.438446 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.438488 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.438829 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.439534 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.441265 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.441783 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.441884 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.441943 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.442164 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.442235 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:55:09.442718 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.437536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.98.2:41711)
0620 08:55:09.438169 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.438181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.438289 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.438612 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.438614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.438639 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.438724 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.438729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.439679 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.439682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.441120 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.441132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.441903 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.441907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.442026 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.442046 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.442501 (+ 455us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.442574 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":108,"thread_start_us":63,"threads_started":1}
I20260620 08:55:09.443147 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6805c9a1fc44df9835a819307bb3bcb" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f05142c3754c4a98d79a771830a80c" is_pre_election: true
I20260620 08:55:09.443207 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6805c9a1fc44df9835a819307bb3bcb" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748" is_pre_election: true
I20260620 08:55:09.443286 8349 raft_consensus.cc:2454] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 0.
I20260620 08:55:09.443449 8071 raft_consensus.cc:2454] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 0.
I20260620 08:55:09.443764 8142 leader_election.cc:302] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:09.444099 8432 raft_consensus.cc:2781] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:09.444209 8432 raft_consensus.cc:482] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:09.444254 8432 raft_consensus.cc:3037] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.445410 8432 raft_consensus.cc:504] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.445746 8432 leader_election.cc:288] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.446102 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6805c9a1fc44df9835a819307bb3bcb" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748"
I20260620 08:55:09.446226 8071 raft_consensus.cc:3037] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.446231 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6805c9a1fc44df9835a819307bb3bcb" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f05142c3754c4a98d79a771830a80c"
I20260620 08:55:09.446357 8349 raft_consensus.cc:3037] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.447090 8071 raft_consensus.cc:2454] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 1.
I20260620 08:55:09.447247 8349 raft_consensus.cc:2454] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 1.
I20260620 08:55:09.447355 8145 leader_election.cc:302] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03daa5caaab64f3a935b90ec6f98b748, 767930edba384d7eb4c849c213b1d3b0; no voters:
I20260620 08:55:09.447512 8432 raft_consensus.cc:2781] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:09.447714 8432 raft_consensus.cc:686] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Running, Role: LEADER
I20260620 08:55:09.447939 8432 consensus_queue.cc:237] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.449903 7910 catalog_manager.cc:5495] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } health_report { overall_health: UNKNOWN } } }
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:09.517120 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.509277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.0.98.62:39373)
0620 08:55:09.509409 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.509414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.509536 (+ 122us) server_negotiation.cc:407] Connection header received
0620 08:55:09.509657 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.509660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.509708 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.509771 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.510385 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.510898 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.511798 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.511994 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.512072 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.514274 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:09.514277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:09.514279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:09.514314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:09.515589 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.516015 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.516018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.516020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.516083 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.516318 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.516321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.516323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.516479 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:09.516560 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.516829 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.516970 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":49}
I20260620 08:55:09.517120 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.509209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39373 (local address 127.0.0.1:39246)
0620 08:55:09.509461 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.509472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.509579 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.509794 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.509797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.510191 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:09.510284 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.510290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.510989 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.510992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.511695 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.511704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.512081 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.512084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.512185 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.512709 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:09.512722 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:09.514106 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:09.515706 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:09.515715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:09.515721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:09.515931 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:09.516168 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:09.516171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:09.516173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:09.516246 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:09.516569 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:09.516575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:09.516684 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.516777 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.516924 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260620 08:55:09.521004 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39246:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:55:09.521693 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:09.526827 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 0977aa83d89d4413a743cbf8c62acdf7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93cef19c756c4311b7d0ed14479ee501]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:55:09.526829 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 0977aa83d89d4413a743cbf8c62acdf7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93cef19c756c4311b7d0ed14479ee501]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:55:09.527213 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0977aa83d89d4413a743cbf8c62acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.527212 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0977aa83d89d4413a743cbf8c62acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.527438 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 0977aa83d89d4413a743cbf8c62acdf7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93cef19c756c4311b7d0ed14479ee501]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:55:09.527853 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0977aa83d89d4413a743cbf8c62acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.529937 8425 tablet_bootstrap.cc:492] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: Bootstrap starting.
I20260620 08:55:09.529965 8426 tablet_bootstrap.cc:492] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: Bootstrap starting.
I20260620 08:55:09.530215 8427 tablet_bootstrap.cc:492] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: Bootstrap starting.
I20260620 08:55:09.530723 8426 tablet_bootstrap.cc:654] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.530766 8425 tablet_bootstrap.cc:654] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.531111 8427 tablet_bootstrap.cc:654] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.531931 8426 tablet_bootstrap.cc:492] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: No bootstrap required, opened a new log
I20260620 08:55:09.532017 8426 ts_tablet_manager.cc:1397] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:55:09.532464 8425 tablet_bootstrap.cc:492] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: No bootstrap required, opened a new log
I20260620 08:55:09.532541 8425 ts_tablet_manager.cc:1397] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 08:55:09.532492 8426 raft_consensus.cc:348] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.532569 8427 tablet_bootstrap.cc:492] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: No bootstrap required, opened a new log
I20260620 08:55:09.532596 8426 raft_consensus.cc:374] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.532625 8426 raft_consensus.cc:729] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.532645 8427 ts_tablet_manager.cc:1397] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:09.532757 8426 consensus_queue.cc:260] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.532918 8425 raft_consensus.cc:348] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.533026 8425 raft_consensus.cc:374] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.533012 8426 ts_tablet_manager.cc:1428] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:55:09.532984 8432 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.533058 8425 raft_consensus.cc:729] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03daa5caaab64f3a935b90ec6f98b748, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.533027 8427 raft_consensus.cc:348] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.533134 8432 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.533141 8427 raft_consensus.cc:374] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.533172 8427 raft_consensus.cc:729] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.533170 8425 consensus_queue.cc:260] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.533281 8427 consensus_queue.cc:260] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.533411 8425 ts_tablet_manager.cc:1428] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:55:09.533511 8427 ts_tablet_manager.cc:1428] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:09.533475 8431 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.533605 8431 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.533567 8447 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.533687 8447 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.549058 8432 raft_consensus.cc:482] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:09.549180 8432 raft_consensus.cc:504] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.549607 8432 leader_election.cc:288] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.549995 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0977aa83d89d4413a743cbf8c62acdf7" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748" is_pre_election: true
I20260620 08:55:09.549995 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0977aa83d89d4413a743cbf8c62acdf7" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f05142c3754c4a98d79a771830a80c" is_pre_election: true
I20260620 08:55:09.550140 8349 raft_consensus.cc:2454] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 0.
I20260620 08:55:09.550159 8071 raft_consensus.cc:2454] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 0.
I20260620 08:55:09.550391 8142 leader_election.cc:302] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03daa5caaab64f3a935b90ec6f98b748, 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:09.550552 8432 raft_consensus.cc:2781] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:09.550606 8432 raft_consensus.cc:482] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:09.550640 8432 raft_consensus.cc:3037] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.551599 8432 raft_consensus.cc:504] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.551939 8432 leader_election.cc:288] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213)
I20260620 08:55:09.552243 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0977aa83d89d4413a743cbf8c62acdf7" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f05142c3754c4a98d79a771830a80c"
I20260620 08:55:09.552250 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0977aa83d89d4413a743cbf8c62acdf7" candidate_uuid: "767930edba384d7eb4c849c213b1d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748"
I20260620 08:55:09.552345 8349 raft_consensus.cc:3037] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.552350 8071 raft_consensus.cc:3037] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.553164 8071 raft_consensus.cc:2454] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 1.
I20260620 08:55:09.553179 8349 raft_consensus.cc:2454] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 767930edba384d7eb4c849c213b1d3b0 in term 1.
I20260620 08:55:09.553414 8142 leader_election.cc:302] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:09.553562 8432 raft_consensus.cc:2781] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:09.553648 8432 raft_consensus.cc:686] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Running, Role: LEADER
I20260620 08:55:09.553761 8432 consensus_queue.cc:237] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.554394 8458 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: TxnStatusTablet state changed. Reason: New leader 767930edba384d7eb4c849c213b1d3b0. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.554494 8458 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:55:09.554976 8470 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:55:09.555154 8470 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:55:09.555387 7912 catalog_manager.cc:5495] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } health_report { overall_health: UNKNOWN } } }
I20260620 08:55:09.570451 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.567149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.0.98.2:40695)
0620 08:55:09.567253 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.567258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.567271 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:09.567469 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.567471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.567520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.567593 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:09.567841 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.568346 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.569154 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.569359 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.569457 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.569702 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.570081 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.570177 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.570259 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.570307 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:55:09.570458 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.567048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40695 (local address 127.0.0.1:47172)
0620 08:55:09.567204 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.567220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.567321 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.567634 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.567637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.567644 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:55:09.567747 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.567753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.568442 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.568445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.569048 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.569056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.569450 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.569454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.569531 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.569575 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.570168 (+ 593us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.570180 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.570239 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.570314 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 08:55:09.573215 8349 raft_consensus.cc:1261] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Refusing update from remote peer 767930edba384d7eb4c849c213b1d3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:09.573231 8071 raft_consensus.cc:1261] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Refusing update from remote peer 767930edba384d7eb4c849c213b1d3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:09.573743 8458 consensus_queue.cc:1035] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:09.573912 8432 consensus_queue.cc:1035] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:09.575446 8447 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: TxnStatusTablet state changed. Reason: New leader 767930edba384d7eb4c849c213b1d3b0. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.575474 8431 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: TxnStatusTablet state changed. Reason: New leader 767930edba384d7eb4c849c213b1d3b0. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.575555 8447 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.575585 8431 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.577349 8458 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.577461 8458 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:55:09.577724 8439 mvcc.cc:205] Tried to move back new op lower bound from 7298849626406490112 to 7298849626333462528. Current Snapshot: MvccSnapshot[applied={T|T < 7298849626406490112}]
I20260620 08:55:09.577832 8447 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.577945 8447 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 03daa5caaab64f3a935b90ec6f98b748: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.579989 8432 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.580087 8432 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 767930edba384d7eb4c849c213b1d3b0: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:55:09.580612 8431 tablet_replica.cc:439] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "767930edba384d7eb4c849c213b1d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } } }
I20260620 08:55:09.580719 8431 tablet_replica.cc:442] T 0977aa83d89d4413a743cbf8c62acdf7 P 89f05142c3754c4a98d79a771830a80c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:55:09.586031 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.582714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.0.98.2:40695)
0620 08:55:09.582818 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.582822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.582865 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:55:09.582997 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.582999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.583035 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.583091 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:09.583307 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.583844 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.584958 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.585189 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.585266 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.585538 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.585604 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.585673 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.585831 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.585889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:55:09.595918 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.592819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55735 (local address 127.0.98.2:40695)
0620 08:55:09.592924 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.592928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.593041 (+ 113us) server_negotiation.cc:407] Connection header received
0620 08:55:09.593157 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.593160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.593197 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.593255 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:09.593481 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.593968 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.594897 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.595103 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.595189 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.595427 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.595487 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.595540 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.595697 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.595780 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":45}
I20260620 08:55:09.595934 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.592709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40695 (local address 127.0.98.2:55735)
0620 08:55:09.592969 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.592980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.593073 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.593265 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.593268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.593274 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:55:09.593395 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.593400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.594058 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.594060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.594793 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.594803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.595196 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.595199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.595281 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.595315 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.595552 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.595557 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.595689 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.595790 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260620 08:55:09.601553 8349 raft_consensus.cc:1261] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Refusing update from remote peer 767930edba384d7eb4c849c213b1d3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:09.601846 8071 raft_consensus.cc:1261] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Refusing update from remote peer 767930edba384d7eb4c849c213b1d3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:09.601934 8458 consensus_queue.cc:1035] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:09.602164 8458 consensus_queue.cc:1035] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:09.606556 8441 mvcc.cc:205] Tried to move back new op lower bound from 7298849626524692480 to 7298849625900937216. Current Snapshot: MvccSnapshot[applied={T|T < 7298849626524692480}]
I20260620 08:55:09.641383 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:09.641592 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:09.644083 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Sending DeleteTablet for 3 replicas of tablet f6805c9a1fc44df9835a819307bb3bcb
I20260620 08:55:09.644927 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet f6805c9a1fc44df9835a819307bb3bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53022
I20260620 08:55:09.645147 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet f6805c9a1fc44df9835a819307bb3bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34462
I20260620 08:55:09.645201 8452 tablet_replica.cc:330] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: stopping tablet replica
I20260620 08:55:09.645320 8452 raft_consensus.cc:2229] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:09.645356 8450 tablet_replica.cc:330] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: stopping tablet replica
I20260620 08:55:09.645442 8452 raft_consensus.cc:2258] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:09.645491 8450 raft_consensus.cc:2229] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:09.645608 8450 raft_consensus.cc:2258] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:09.645793 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet f6805c9a1fc44df9835a819307bb3bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260620 08:55:09.645959 8451 tablet_replica.cc:330] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: stopping tablet replica
I20260620 08:55:09.646072 8451 raft_consensus.cc:2229] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:09.646449 8452 ts_tablet_manager.cc:1905] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:09.646449 8451 raft_consensus.cc:2258] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:09.647217 8451 ts_tablet_manager.cc:1905] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:09.649273 8452 ts_tablet_manager.cc:1918] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:55:09.649782 8451 ts_tablet_manager.cc:1918] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:55:09.649832 8451 log.cc:1196] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/f6805c9a1fc44df9835a819307bb3bcb
I20260620 08:55:09.649799 8450 ts_tablet_manager.cc:1905] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:09.649984 8451 ts_tablet_manager.cc:1939] T f6805c9a1fc44df9835a819307bb3bcb P 767930edba384d7eb4c849c213b1d3b0: Deleting consensus metadata
I20260620 08:55:09.650874 7899 catalog_manager.cc:4841] TS 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695): tablet f6805c9a1fc44df9835a819307bb3bcb (table test-table [id=979047c1421640959ecdc922f55c6ed2]) successfully deleted
I20260620 08:55:09.649358 8452 log.cc:1196] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/f6805c9a1fc44df9835a819307bb3bcb
I20260620 08:55:09.651887 8452 ts_tablet_manager.cc:1939] T f6805c9a1fc44df9835a819307bb3bcb P 89f05142c3754c4a98d79a771830a80c: Deleting consensus metadata
I20260620 08:55:09.652420 8450 ts_tablet_manager.cc:1918] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:55:09.652498 8450 log.cc:1196] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/f6805c9a1fc44df9835a819307bb3bcb
I20260620 08:55:09.652585 7898 catalog_manager.cc:4841] TS 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213): tablet f6805c9a1fc44df9835a819307bb3bcb (table test-table [id=979047c1421640959ecdc922f55c6ed2]) successfully deleted
I20260620 08:55:09.652719 8450 ts_tablet_manager.cc:1939] T f6805c9a1fc44df9835a819307bb3bcb P 03daa5caaab64f3a935b90ec6f98b748: Deleting consensus metadata
I20260620 08:55:09.653436 7901 catalog_manager.cc:4841] TS 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223): tablet f6805c9a1fc44df9835a819307bb3bcb (table test-table [id=979047c1421640959ecdc922f55c6ed2]) successfully deleted
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:55:09.673242 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:09.666299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.0.98.62:39373)
0620 08:55:09.666410 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.666414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.666609 (+ 195us) server_negotiation.cc:407] Connection header received
0620 08:55:09.666743 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.666747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.666788 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.666855 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.667585 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.668182 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.669382 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.669586 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.669684 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.673141 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39258: BlockingRecv error: recv got EOF from 127.0.0.1:39258 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 08:55:09 dist-test-slave-fb1v krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 08:55:09.684028 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.675455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.0.98.62:39373)
0620 08:55:09.675575 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.675579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.675814 (+ 235us) server_negotiation.cc:407] Connection header received
0620 08:55:09.675901 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.675902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.675940 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.676003 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:09.676667 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.677180 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.678256 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.678467 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.678544 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.681167 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:09.681170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:09.681172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:09.681195 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:09.682426 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.682878 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.682881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.682882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.682949 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:09.683199 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:09.683201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:09.683202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:09.683358 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:09.683446 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.683786 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.683878 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:55:09.686450 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:55:09.686930 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:09.691893 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 863b98f8b7d44fff8873fb7c0dcf34d2 (DEFAULT_TABLE table=test-table [id=2358ee7382624e04802ae0973a59e0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.691898 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 863b98f8b7d44fff8873fb7c0dcf34d2 (DEFAULT_TABLE table=test-table [id=2358ee7382624e04802ae0973a59e0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.692184 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863b98f8b7d44fff8873fb7c0dcf34d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.692184 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863b98f8b7d44fff8873fb7c0dcf34d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.694306 8425 tablet_bootstrap.cc:492] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Bootstrap starting.
I20260620 08:55:09.694569 8426 tablet_bootstrap.cc:492] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Bootstrap starting.
I20260620 08:55:09.695036 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 863b98f8b7d44fff8873fb7c0dcf34d2 (DEFAULT_TABLE table=test-table [id=2358ee7382624e04802ae0973a59e0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:09.695181 8425 tablet_bootstrap.cc:654] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.695281 8426 tablet_bootstrap.cc:654] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.695304 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863b98f8b7d44fff8873fb7c0dcf34d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:09.696373 8425 tablet_bootstrap.cc:492] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: No bootstrap required, opened a new log
I20260620 08:55:09.696465 8425 ts_tablet_manager.cc:1397] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 08:55:09.696879 8425 raft_consensus.cc:348] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.696990 8425 raft_consensus.cc:374] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.697017 8425 raft_consensus.cc:729] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03daa5caaab64f3a935b90ec6f98b748, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.697134 8425 consensus_queue.cc:260] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.697391 8425 ts_tablet_manager.cc:1428] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:09.697625 8427 tablet_bootstrap.cc:492] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Bootstrap starting.
I20260620 08:55:09.698385 8427 tablet_bootstrap.cc:654] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:09.699347 8427 tablet_bootstrap.cc:492] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: No bootstrap required, opened a new log
I20260620 08:55:09.699412 8427 ts_tablet_manager.cc:1397] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:09.699784 8427 raft_consensus.cc:348] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.699873 8427 raft_consensus.cc:374] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.699894 8427 raft_consensus.cc:729] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.699993 8427 consensus_queue.cc:260] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.700230 8427 ts_tablet_manager.cc:1428] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:55:09.701270 8426 tablet_bootstrap.cc:492] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: No bootstrap required, opened a new log
I20260620 08:55:09.701364 8426 ts_tablet_manager.cc:1397] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 08:55:09.701817 8426 raft_consensus.cc:348] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.701930 8426 raft_consensus.cc:374] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:09.701969 8426 raft_consensus.cc:729] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Initialized, Role: FOLLOWER
I20260620 08:55:09.702090 8426 consensus_queue.cc:260] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.702419 8426 ts_tablet_manager.cc:1428] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:55:09.925851 8431 raft_consensus.cc:482] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:09.926064 8431 raft_consensus.cc:504] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.928328 8431 leader_election.cc:288] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695)
I20260620 08:55:09.932869 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.928778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40699 (local address 127.0.98.2:40695)
0620 08:55:09.928916 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.928921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.929257 (+ 336us) server_negotiation.cc:407] Connection header received
0620 08:55:09.929475 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.929478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.929525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.929590 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.929809 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.930557 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.931905 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.932300 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.932414 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.932477 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.932677 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.932748 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:55:09.933169 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.928573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40695 (local address 127.0.98.3:40699)
0620 08:55:09.929175 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.929187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.929272 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.929602 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.929604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.929621 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.929720 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.929726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.930683 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.930687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.931804 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.931810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.932427 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.932430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.932540 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.932564 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.932633 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.932707 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":65,"thread_start_us":66,"threads_started":1}
I20260620 08:55:09.933235 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.928927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38229 (local address 127.0.98.1:36223)
0620 08:55:09.929044 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.929048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.929077 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:55:09.929168 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.929171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.929218 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.929305 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:09.929597 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.930176 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.931748 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.932355 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.932449 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.932523 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.933040 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.933120 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:55:09.933605 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.928573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.98.3:38229)
0620 08:55:09.928919 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:09.928933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:09.929039 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:09.929335 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:09.929338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:09.929354 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:09.929490 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.929496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.930355 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.930359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.931610 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:09.931622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.932578 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.932581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:09.932667 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.932686 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:55:09.933350 (+ 664us) client_negotiation.cc:240] Negotiation successful
0620 08:55:09.933423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":73,"threads_started":1}
I20260620 08:55:09.933908 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b98f8b7d44fff8873fb7c0dcf34d2" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767930edba384d7eb4c849c213b1d3b0" is_pre_election: true
I20260620 08:55:09.934070 8211 raft_consensus.cc:2454] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 0.
I20260620 08:55:09.934125 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b98f8b7d44fff8873fb7c0dcf34d2" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748" is_pre_election: true
I20260620 08:55:09.934235 8071 raft_consensus.cc:2454] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 0.
I20260620 08:55:09.934418 8283 leader_election.cc:302] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:09.934659 8431 raft_consensus.cc:2781] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:09.934727 8431 raft_consensus.cc:482] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:09.934767 8431 raft_consensus.cc:3037] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.935968 8431 raft_consensus.cc:504] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.936336 8431 leader_election.cc:288] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 election: Requested vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695)
I20260620 08:55:09.936671 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b98f8b7d44fff8873fb7c0dcf34d2" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767930edba384d7eb4c849c213b1d3b0"
I20260620 08:55:09.936797 8211 raft_consensus.cc:3037] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.936853 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b98f8b7d44fff8873fb7c0dcf34d2" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748"
I20260620 08:55:09.936964 8071 raft_consensus.cc:3037] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:09.937920 8211 raft_consensus.cc:2454] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 1.
I20260620 08:55:09.937956 8071 raft_consensus.cc:2454] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 1.
I20260620 08:55:09.938179 8283 leader_election.cc:302] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:09.938330 8431 raft_consensus.cc:2781] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:09.938498 8431 raft_consensus.cc:686] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 1 LEADER]: Becoming Leader. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Running, Role: LEADER
I20260620 08:55:09.938736 8431 consensus_queue.cc:237] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:09.941094 7914 catalog_manager.cc:5495] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f05142c3754c4a98d79a771830a80c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "89f05142c3754c4a98d79a771830a80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } health_report { overall_health: HEALTHY } } }
I20260620 08:55:09.998603 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:09.994679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.98.3:38213)
0620 08:55:09.994932 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:09.994937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:09.994950 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:09.995034 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:09.995036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:09.995089 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:09.995193 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:09.995553 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.996308 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.997225 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:09.997485 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:09.997576 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:09.997783 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:09.998245 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:09.998329 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:09.998426 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 08:55:09.998476 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260620 08:55:10.002118 8071 raft_consensus.cc:1261] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Refusing update from remote peer 89f05142c3754c4a98d79a771830a80c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:10.002118 8211 raft_consensus.cc:1261] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 89f05142c3754c4a98d79a771830a80c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:10.002841 8499 consensus_queue.cc:1035] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [LEADER]: Connected to new peer: Peer: permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:10.003057 8431 consensus_queue.cc:1035] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:10.015980 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:10.016204 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:10.018728 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Sending DeleteTablet for 3 replicas of tablet 863b98f8b7d44fff8873fb7c0dcf34d2
I20260620 08:55:10.019593 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 863b98f8b7d44fff8873fb7c0dcf34d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53022
I20260620 08:55:10.019593 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 863b98f8b7d44fff8873fb7c0dcf34d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34462
I20260620 08:55:10.019742 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 863b98f8b7d44fff8873fb7c0dcf34d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260620 08:55:10.019879 8452 tablet_replica.cc:330] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: stopping tablet replica
I20260620 08:55:10.019879 8450 tablet_replica.cc:330] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: stopping tablet replica
I20260620 08:55:10.019979 8451 tablet_replica.cc:330] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: stopping tablet replica
I20260620 08:55:10.020025 8450 raft_consensus.cc:2229] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:10.020025 8452 raft_consensus.cc:2229] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:10.020097 8451 raft_consensus.cc:2229] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:10.020244 8451 raft_consensus.cc:2258] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:10.020252 8450 raft_consensus.cc:2258] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:10.020502 8452 raft_consensus.cc:2258] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:10.021354 8450 ts_tablet_manager.cc:1905] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:10.021358 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:55:10.021528 8452 ts_tablet_manager.cc:1905] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:55:10.021857 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:10.024133 8452 ts_tablet_manager.cc:1918] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:10.024209 8452 log.cc:1196] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/863b98f8b7d44fff8873fb7c0dcf34d2
I20260620 08:55:10.024432 8452 ts_tablet_manager.cc:1939] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 89f05142c3754c4a98d79a771830a80c: Deleting consensus metadata
I20260620 08:55:10.024895 8450 ts_tablet_manager.cc:1918] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:10.024964 8450 log.cc:1196] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/863b98f8b7d44fff8873fb7c0dcf34d2
I20260620 08:55:10.025229 8450 ts_tablet_manager.cc:1939] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 03daa5caaab64f3a935b90ec6f98b748: Deleting consensus metadata
I20260620 08:55:10.025745 8451 ts_tablet_manager.cc:1905] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:10.025967 7901 catalog_manager.cc:4841] TS 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223): tablet 863b98f8b7d44fff8873fb7c0dcf34d2 (table test-table [id=2358ee7382624e04802ae0973a59e0c7]) successfully deleted
I20260620 08:55:10.028008 8451 ts_tablet_manager.cc:1918] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:10.028059 8451 log.cc:1196] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/863b98f8b7d44fff8873fb7c0dcf34d2
I20260620 08:55:10.028209 8451 ts_tablet_manager.cc:1939] T 863b98f8b7d44fff8873fb7c0dcf34d2 P 767930edba384d7eb4c849c213b1d3b0: Deleting consensus metadata
I20260620 08:55:10.028585 7898 catalog_manager.cc:4841] TS 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213): tablet 863b98f8b7d44fff8873fb7c0dcf34d2 (table test-table [id=2358ee7382624e04802ae0973a59e0c7]) successfully deleted
I20260620 08:55:10.028836 7899 catalog_manager.cc:4841] TS 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695): tablet 863b98f8b7d44fff8873fb7c0dcf34d2 (table test-table [id=2358ee7382624e04802ae0973a59e0c7]) successfully deleted
I20260620 08:55:10.029728 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 81c2b421f24e45e2b28b4a7854ca6eba (DEFAULT_TABLE table=test-table [id=fbfefb013b7243a0a00c950186e4c587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:10.029975 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2b421f24e45e2b28b4a7854ca6eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:10.029928 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 81c2b421f24e45e2b28b4a7854ca6eba (DEFAULT_TABLE table=test-table [id=fbfefb013b7243a0a00c950186e4c587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:10.030130 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2b421f24e45e2b28b4a7854ca6eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:10.032075 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 81c2b421f24e45e2b28b4a7854ca6eba (DEFAULT_TABLE table=test-table [id=fbfefb013b7243a0a00c950186e4c587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:10.032249 8427 tablet_bootstrap.cc:492] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Bootstrap starting.
I20260620 08:55:10.032305 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c2b421f24e45e2b28b4a7854ca6eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:10.032330 8425 tablet_bootstrap.cc:492] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748: Bootstrap starting.
I20260620 08:55:10.033092 8427 tablet_bootstrap.cc:654] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:10.033088 8425 tablet_bootstrap.cc:654] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:10.034327 8427 tablet_bootstrap.cc:492] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: No bootstrap required, opened a new log
I20260620 08:55:10.034365 8426 tablet_bootstrap.cc:492] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Bootstrap starting.
I20260620 08:55:10.034380 8425 tablet_bootstrap.cc:492] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748: No bootstrap required, opened a new log
I20260620 08:55:10.034407 8427 ts_tablet_manager.cc:1397] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:55:10.034442 8425 ts_tablet_manager.cc:1397] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:10.034801 8425 raft_consensus.cc:348] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.034801 8427 raft_consensus.cc:348] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.034901 8425 raft_consensus.cc:374] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:10.034900 8427 raft_consensus.cc:374] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:10.034932 8427 raft_consensus.cc:729] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Initialized, Role: FOLLOWER
I20260620 08:55:10.034932 8425 raft_consensus.cc:729] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03daa5caaab64f3a935b90ec6f98b748, State: Initialized, Role: FOLLOWER
I20260620 08:55:10.035041 8425 consensus_queue.cc:260] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.035041 8427 consensus_queue.cc:260] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.035174 8426 tablet_bootstrap.cc:654] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:10.035290 8425 ts_tablet_manager.cc:1428] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:55:10.035290 8427 ts_tablet_manager.cc:1428] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:55:10.036603 8426 tablet_bootstrap.cc:492] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: No bootstrap required, opened a new log
I20260620 08:55:10.036700 8426 ts_tablet_manager.cc:1397] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 08:55:10.037081 8426 raft_consensus.cc:348] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.037189 8426 raft_consensus.cc:374] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:10.037220 8426 raft_consensus.cc:729] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767930edba384d7eb4c849c213b1d3b0, State: Initialized, Role: FOLLOWER
I20260620 08:55:10.037343 8426 consensus_queue.cc:260] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.037585 8426 ts_tablet_manager.cc:1428] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:55:10.259150 8431 raft_consensus.cc:482] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:10.259305 8431 raft_consensus.cc:504] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.259745 8431 leader_election.cc:288] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695)
I20260620 08:55:10.260351 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2b421f24e45e2b28b4a7854ca6eba" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767930edba384d7eb4c849c213b1d3b0" is_pre_election: true
I20260620 08:55:10.260351 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2b421f24e45e2b28b4a7854ca6eba" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748" is_pre_election: true
I20260620 08:55:10.260504 8071 raft_consensus.cc:2454] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 0.
I20260620 08:55:10.260504 8211 raft_consensus.cc:2454] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 0.
I20260620 08:55:10.260953 8285 leader_election.cc:302] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03daa5caaab64f3a935b90ec6f98b748, 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:10.261166 8431 raft_consensus.cc:2781] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:10.261233 8431 raft_consensus.cc:482] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:10.261267 8431 raft_consensus.cc:3037] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:10.262188 8431 raft_consensus.cc:504] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.262575 8431 leader_election.cc:288] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 election: Requested vote from peers 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223), 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695)
I20260620 08:55:10.262975 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2b421f24e45e2b28b4a7854ca6eba" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03daa5caaab64f3a935b90ec6f98b748"
I20260620 08:55:10.262988 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c2b421f24e45e2b28b4a7854ca6eba" candidate_uuid: "89f05142c3754c4a98d79a771830a80c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767930edba384d7eb4c849c213b1d3b0"
I20260620 08:55:10.263095 8211 raft_consensus.cc:3037] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:10.263104 8071 raft_consensus.cc:3037] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:10.264081 8071 raft_consensus.cc:2454] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 1.
I20260620 08:55:10.264072 8211 raft_consensus.cc:2454] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f05142c3754c4a98d79a771830a80c in term 1.
I20260620 08:55:10.264382 8283 leader_election.cc:302] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03daa5caaab64f3a935b90ec6f98b748, 767930edba384d7eb4c849c213b1d3b0, 89f05142c3754c4a98d79a771830a80c; no voters:
I20260620 08:55:10.264539 8431 raft_consensus.cc:2781] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:10.264640 8431 raft_consensus.cc:686] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 1 LEADER]: Becoming Leader. State: Replica: 89f05142c3754c4a98d79a771830a80c, State: Running, Role: LEADER
I20260620 08:55:10.264782 8431 consensus_queue.cc:237] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } }
I20260620 08:55:10.266372 7912 catalog_manager.cc:5495] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f05142c3754c4a98d79a771830a80c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "89f05142c3754c4a98d79a771830a80c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f05142c3754c4a98d79a771830a80c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38213 } health_report { overall_health: HEALTHY } } }
I20260620 08:55:10.342381 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:10.339070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40695 (local address 127.0.98.3:34609)
0620 08:55:10.339359 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:10.339371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:10.339462 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:10.339680 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:10.339682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:10.339689 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:55:10.339779 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:10.339783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.340445 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.340447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:10.341057 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:10.341064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.341473 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.341476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:10.341541 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:10.341581 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:55:10.342117 (+ 536us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:55:10.342123 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:55:10.342188 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:55:10.342257 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260620 08:55:10.342522 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:10.339220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34609 (local address 127.0.98.2:40695)
0620 08:55:10.339362 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:10.339366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:10.339450 (+ 84us) server_negotiation.cc:407] Connection header received
0620 08:55:10.339546 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:10.339548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:10.339588 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:10.339665 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:10.339855 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.340367 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.341171 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.341381 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.341455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:10.341864 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:10.341992 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:10.342094 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:10.342336 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 08:55:10.342389 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:55:10.348965 8211 raft_consensus.cc:1261] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 89f05142c3754c4a98d79a771830a80c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:10.349048 8071 raft_consensus.cc:1261] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Refusing update from remote peer 89f05142c3754c4a98d79a771830a80c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:10.349398 8505 consensus_queue.cc:1035] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [LEADER]: Connected to new peer: Peer: permanent_uuid: "767930edba384d7eb4c849c213b1d3b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:10.349561 8431 consensus_queue.cc:1035] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03daa5caaab64f3a935b90ec6f98b748" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:10.364588 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:10.361371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34301 (local address 127.0.98.3:38213)
0620 08:55:10.361488 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:10.361492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:10.361503 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:55:10.361539 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:10.361542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:10.361588 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:10.361663 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:10.361903 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.362409 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.363280 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.363601 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.363715 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:10.364006 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:10.364073 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:10.364142 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:10.364438 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 08:55:10.364486 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:10.364740 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:10.361066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38213 (local address 127.0.98.2:34301)
0620 08:55:10.361336 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:10.361350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:10.361439 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:10.361687 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:10.361689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:10.361698 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:55:10.361789 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:10.361794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.362550 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.362556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:10.363190 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:10.363199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:10.363790 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:10.363793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:10.363856 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:10.363883 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:55:10.364262 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:55:10.364278 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:55:10.364410 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 08:55:10.364539 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20260620 08:55:10.382184 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:10.382333 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:10.384168 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be5727c5946443aea20a7726b5df57bb: Sending DeleteTablet for 3 replicas of tablet 81c2b421f24e45e2b28b4a7854ca6eba
I20260620 08:55:10.384707 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c2b421f24e45e2b28b4a7854ca6eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53022
I20260620 08:55:10.384718 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c2b421f24e45e2b28b4a7854ca6eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34462
I20260620 08:55:10.384891 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c2b421f24e45e2b28b4a7854ca6eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260620 08:55:10.385085 8450 tablet_replica.cc:330] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748: stopping tablet replica
I20260620 08:55:10.385232 8450 raft_consensus.cc:2229] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:10.385352 8450 raft_consensus.cc:2258] T 81c2b421f24e45e2b28b4a7854ca6eba P 03daa5caaab64f3a935b90ec6f98b748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:10.385404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 7986
I20260620 08:55:10.385524 8452 tablet_replica.cc:330] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: stopping tablet replica
I20260620 08:55:10.385636 8452 raft_consensus.cc:2229] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:10.385805 8452 raft_consensus.cc:2258] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:10.385943 8451 tablet_replica.cc:330] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: stopping tablet replica
I20260620 08:55:10.386056 8451 raft_consensus.cc:2229] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:10.386169 8451 raft_consensus.cc:2258] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:10.386933 8451 ts_tablet_manager.cc:1905] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:10.387639 8452 ts_tablet_manager.cc:1905] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:10.389319 8451 ts_tablet_manager.cc:1918] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:55:10.389397 8451 log.cc:1196] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/81c2b421f24e45e2b28b4a7854ca6eba
I20260620 08:55:10.389636 8451 ts_tablet_manager.cc:1939] T 81c2b421f24e45e2b28b4a7854ca6eba P 767930edba384d7eb4c849c213b1d3b0: Deleting consensus metadata
I20260620 08:55:10.389901 8452 ts_tablet_manager.cc:1918] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:55:10.389976 8452 log.cc:1196] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/81c2b421f24e45e2b28b4a7854ca6eba
I20260620 08:55:10.390211 8452 ts_tablet_manager.cc:1939] T 81c2b421f24e45e2b28b4a7854ca6eba P 89f05142c3754c4a98d79a771830a80c: Deleting consensus metadata
I20260620 08:55:10.390444 7899 catalog_manager.cc:4841] TS 767930edba384d7eb4c849c213b1d3b0 (127.0.98.2:40695): tablet 81c2b421f24e45e2b28b4a7854ca6eba (table test-table [id=fbfefb013b7243a0a00c950186e4c587]) successfully deleted
I20260620 08:55:10.390897 7898 catalog_manager.cc:4841] TS 89f05142c3754c4a98d79a771830a80c (127.0.98.3:38213): tablet 81c2b421f24e45e2b28b4a7854ca6eba (table test-table [id=fbfefb013b7243a0a00c950186e4c587]) successfully deleted
W20260620 08:55:10.393203 7901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36223 (error 108)
I20260620 08:55:10.393887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 8126
I20260620 08:55:10.394215 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:10.393824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.0.1:44718)
0620 08:55:10.394038 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:10.394116 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
W20260620 08:55:10.394299 7901 catalog_manager.cc:4584] TS 03daa5caaab64f3a935b90ec6f98b748 (127.0.98.1:36223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81c2b421f24e45e2b28b4a7854ca6eba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36223: connect: Connection refused (error 111)
I20260620 08:55:10.400517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 8266
I20260620 08:55:10.406159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 7882
2026-06-20T08:55:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3257 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:10 dist-test-slave-fb1v krb5kdc[8526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:10 dist-test-slave-fb1v krb5kdc[8526](info): set up 2 sockets
Jun 20 08:55:10 dist-test-slave-fb1v krb5kdc[8526](info): commencing operation
krb5kdc: starting...
W20260620 08:55:11.376099 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:11 dist-test-slave-fb1v krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:55:11.495220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:11.608040 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:11.608330 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:11.608369 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:11.612398 8539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:11.612468 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:11.612488 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:11.612505 8539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:11.612519 8539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:11.617877 8539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:11.618867 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:11.619843 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:11.628100 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:11.628435 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:11.628611 8539 server_base.cc:1034] running on GCE node
W20260620 08:55:11.628712 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:11.629328 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:11.630399 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:11.631567 8539 hybrid_clock.cc:648] HybridClock initialized: now 1781945711631534 us; error 52 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T08:55:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1238 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:11 dist-test-slave-fb1v krb5kdc[8552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:11 dist-test-slave-fb1v krb5kdc[8552](info): set up 2 sockets
Jun 20 08:55:11 dist-test-slave-fb1v krb5kdc[8552](info): commencing operation
krb5kdc: starting...
W20260620 08:55:12.615996 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:12 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:55:12.734470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:12.838723 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:12.838960 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:12.838999 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:12.842960 8565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:12.843022 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:12.843044 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:12.843068 8565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:12.843088 8565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:12.848217 8565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:12.849123 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:12.850013 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:12.858707 8570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:12.858879 8565 server_base.cc:1034] running on GCE node
W20260620 08:55:12.858700 8573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:12.858700 8571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:12.859537 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:12.860584 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:12.861759 8565 hybrid_clock.cc:648] HybridClock initialized: now 1781945712861731 us; error 38 us; skew 500 ppm
Jun 20 08:55:12 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:12.864473 8565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:55:12.865335 8565 webserver.cc:460] Webserver started at http://127.0.98.62:37101/ using document root <none> and password file <none>
I20260620 08:55:12.865859 8565 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:12.865916 8565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:12.866098 8565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:12.868062 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "99cd76fe589041dca97c14dcd9f59e98"
format_stamp: "Formatted at 2026-06-20 08:55:12 on dist-test-slave-fb1v"
server_key: "8e91cc68187b1ed41999f6de606f5fd9"
server_key_iv: "fb8d402ea370172808a4d44975e4821d"
server_key_version: "encryptionkey@0"
I20260620 08:55:12.868557 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "99cd76fe589041dca97c14dcd9f59e98"
format_stamp: "Formatted at 2026-06-20 08:55:12 on dist-test-slave-fb1v"
server_key: "8e91cc68187b1ed41999f6de606f5fd9"
server_key_iv: "fb8d402ea370172808a4d44975e4821d"
server_key_version: "encryptionkey@0"
I20260620 08:55:12.872015 8565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:55:12.874207 8580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:12.875334 8565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:12.875434 8565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "99cd76fe589041dca97c14dcd9f59e98"
format_stamp: "Formatted at 2026-06-20 08:55:12 on dist-test-slave-fb1v"
server_key: "8e91cc68187b1ed41999f6de606f5fd9"
server_key_iv: "fb8d402ea370172808a4d44975e4821d"
server_key_version: "encryptionkey@0"
I20260620 08:55:12.875531 8565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:12.893481 8565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:12.896474 8565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:12.896643 8565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:12.903196 8565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45335
I20260620 08:55:12.903205 8632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45335 every 8 connection(s)
I20260620 08:55:12.904085 8565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:12.906702 8633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:12.910034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 8565
I20260620 08:55:12.910156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:12.910409 392 external_mini_cluster.cc:1428] Setting key a4bbe642325134fe33b3dcf44a4575f3
I20260620 08:55:12.912448 8633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Bootstrap starting.
I20260620 08:55:12.915042 8633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:12.915892 8633 log.cc:824] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:12.918058 8633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: No bootstrap required, opened a new log
Jun 20 08:55:12 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:12.922220 8633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:12.922508 8633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:12.922600 8633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cd76fe589041dca97c14dcd9f59e98, State: Initialized, Role: FOLLOWER
I20260620 08:55:12.923091 8633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:12.923249 8633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:12.923313 8633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:12.923401 8633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:12.924746 8633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:12.925123 8633 leader_election.cc:302] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99cd76fe589041dca97c14dcd9f59e98; no voters:
I20260620 08:55:12.925462 8633 leader_election.cc:288] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:12.925557 8638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:12.925755 8638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 LEADER]: Becoming Leader. State: Replica: 99cd76fe589041dca97c14dcd9f59e98, State: Running, Role: LEADER
I20260620 08:55:12.925904 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:12.911786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.0.98.62:45335)
0620 08:55:12.912201 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:12.912218 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:12.912266 (+ 48us) server_negotiation.cc:407] Connection header received
0620 08:55:12.913069 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:12.913091 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:12.913474 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:12.913859 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:12.914504 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:12.915977 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:12.917007 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:12.917681 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:12.917849 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:12.920544 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:12.920551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:12.920569 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:12.920605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:12.922481 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:12.922965 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:12.922971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:12.922976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:12.923111 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:12.923422 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:12.923425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:12.923427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:12.924106 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:12.924327 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:12.924605 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 08:55:12.924909 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":278,"thread_start_us":99,"threads_started":1}
I20260620 08:55:12.926085 8638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:12.926661 8633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:12.929222 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99cd76fe589041dca97c14dcd9f59e98. Latest consensus state: current_term: 1 leader_uuid: "99cd76fe589041dca97c14dcd9f59e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } } }
I20260620 08:55:12.929317 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:12.928623 8639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99cd76fe589041dca97c14dcd9f59e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } } }
I20260620 08:55:12.929499 8639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: This master's current role is: LEADER
W20260620 08:55:12.931063 8652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:55:12.931129 8652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:55:12.931209 8653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:12.932140 8653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:12.937657 8653 catalog_manager.cc:1347] Generated new cluster ID: 94499d82dddb46f088eb620cbda65f4b
I20260620 08:55:12.937711 8653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:12.958621 8653 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:12.959592 8653 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:12.966310 8653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Generated new TSK 0
I20260620 08:55:12.967176 8653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:55:13.011502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45335
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:13.122901 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:13.123153 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:13.123194 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:13.127259 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:13.127319 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:13.127377 8659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:13.132676 8659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:13.133641 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:13.134564 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:13.144160 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.144361 8659 server_base.cc:1034] running on GCE node
W20260620 08:55:13.144160 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:13.144163 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.144861 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:13.145460 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:13.146638 8659 hybrid_clock.cc:648] HybridClock initialized: now 1781945713146609 us; error 40 us; skew 500 ppm
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:13.149394 8659 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:55:13.150243 8659 webserver.cc:460] Webserver started at http://127.0.98.1:35803/ using document root <none> and password file <none>
I20260620 08:55:13.150846 8659 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:13.150928 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:13.151117 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:13.153115 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "67fee44869394924b15527033cb52049"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "d10b907eef5800c5358f7260038c1d2c"
server_key_iv: "72884c131b01a90414e33087b02032bc"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.153630 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "67fee44869394924b15527033cb52049"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "d10b907eef5800c5358f7260038c1d2c"
server_key_iv: "72884c131b01a90414e33087b02032bc"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.157229 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 08:55:13.159700 8674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.161015 8659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:55:13.161126 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "67fee44869394924b15527033cb52049"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "d10b907eef5800c5358f7260038c1d2c"
server_key_iv: "72884c131b01a90414e33087b02032bc"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.161224 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:13.187476 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:13.190552 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:13.190749 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:13.191324 8659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:13.192221 8659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:13.192270 8659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.192322 8659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:13.192348 8659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.200840 8659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36299
I20260620 08:55:13.200937 8787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36299 every 8 connection(s)
I20260620 08:55:13.201704 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:13.209057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 8659
I20260620 08:55:13.209213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:13.209492 392 external_mini_cluster.cc:1428] Setting key fb21ba54c5722aef1fa5584a29a63706
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:13.215749 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:13.203593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40211 (local address 127.0.98.62:45335)
0620 08:55:13.203750 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:13.203754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:13.204563 (+ 809us) server_negotiation.cc:407] Connection header received
0620 08:55:13.205866 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:13.205869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:13.205916 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:13.205989 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:13.207098 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.207681 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.208874 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.209116 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.209223 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:13.212367 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:13.212371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:13.212374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:13.212400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:13.213760 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:13.214258 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:13.214261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:13.214263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:13.214342 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:13.214671 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:13.214675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:13.214676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:13.214852 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:13.214935 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:13.215422 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 08:55:13.215551 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 08:55:13.216562 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:13.203944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:40211)
0620 08:55:13.204414 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:13.204456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:13.205669 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:13.206131 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:13.206140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:13.206477 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:13.206950 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:13.206962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.207815 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.207818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:13.208678 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:13.208687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.209288 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.209292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:13.209446 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:13.210324 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:13.210350 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:13.212198 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:13.213870 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:13.213876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:13.213892 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:13.214167 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:13.214423 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:13.214426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:13.214428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:13.214575 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:13.214948 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:13.214955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:13.215203 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 08:55:13.215370 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 08:55:13.215737 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
I20260620 08:55:13.217682 8788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45335
I20260620 08:55:13.217911 8788 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:55:13.218415 8788 heartbeater.cc:511] Master 127.0.98.62:45335 requested a full tablet report, sending...
I20260620 08:55:13.219887 8597 ts_manager.cc:194] Registered new tserver with Master: 67fee44869394924b15527033cb52049 (127.0.98.1:36299)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:55:13.221280 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:55:13.236954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45335
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:13.345873 8793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:13.346165 8793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:13.346228 8793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:13.350850 8793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:13.350930 8793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:13.351015 8793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:13.358217 8793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:13.359283 8793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:13.360342 8793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:55:13.369617 8793 server_base.cc:1034] running on GCE node
W20260620 08:55:13.369639 8801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:13.369617 8798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:13.369635 8799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.370260 8793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:13.370860 8793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:13.372041 8793 hybrid_clock.cc:648] HybridClock initialized: now 1781945713371998 us; error 55 us; skew 500 ppm
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:13.374666 8793 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:55:13.375491 8793 webserver.cc:460] Webserver started at http://127.0.98.2:36959/ using document root <none> and password file <none>
I20260620 08:55:13.376070 8793 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:13.376137 8793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:13.376315 8793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:13.378290 8793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "2de537629ddf4daa96ccd0e89e077da3"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "665fb70112907411493e2b25150d39f0"
server_key_iv: "8d0c93aa3f15051e1236e78e34a76e17"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.378818 8793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2de537629ddf4daa96ccd0e89e077da3"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "665fb70112907411493e2b25150d39f0"
server_key_iv: "8d0c93aa3f15051e1236e78e34a76e17"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.382383 8793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 08:55:13.385075 8808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.386330 8793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:13.386456 8793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "2de537629ddf4daa96ccd0e89e077da3"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "665fb70112907411493e2b25150d39f0"
server_key_iv: "8d0c93aa3f15051e1236e78e34a76e17"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.386566 8793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:13.415019 8793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:13.418177 8793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:13.418365 8793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:13.418934 8793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:13.419780 8793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:13.419832 8793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.419885 8793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:13.419910 8793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.429032 8793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38035
I20260620 08:55:13.429065 8921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38035 every 8 connection(s)
I20260620 08:55:13.429955 8793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:13.433533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 8793
I20260620 08:55:13.433629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:55:13.433885 392 external_mini_cluster.cc:1428] Setting key 4c759d2b38ba5e3b6314010f3f2713da
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:55:13.443320 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:13.431664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57475 (local address 127.0.98.62:45335)
0620 08:55:13.431788 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:13.431792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:13.432569 (+ 777us) server_negotiation.cc:407] Connection header received
0620 08:55:13.433923 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:13.433926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:13.433980 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:13.434055 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:13.435286 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.435870 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.437261 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.437484 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.437563 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:13.440133 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:13.440136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:13.440138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:13.440161 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:13.441389 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:13.441841 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:13.441844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:13.441846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:13.441917 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:13.442253 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:13.442256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:13.442258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:13.442428 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:13.442510 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:13.443000 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 08:55:13.443162 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:55:13.444172 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:13.432023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:57475)
0620 08:55:13.432429 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:13.432469 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:13.433737 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:13.434234 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:13.434241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:13.434644 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:13.435166 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:13.435182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.435980 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.435983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:13.437122 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:13.437133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.437623 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.437626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:13.437800 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:13.438374 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:13.438393 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:13.439948 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:13.441493 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:13.441499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:13.441513 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:13.441751 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:13.442000 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:13.442003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:13.442005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:13.442167 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:13.442521 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:13.442527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:13.442782 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 08:55:13.442947 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 08:55:13.443262 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:55:13.445459 8922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45335
I20260620 08:55:13.445772 8922 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:13.446281 8922 heartbeater.cc:511] Master 127.0.98.62:45335 requested a full tablet report, sending...
I20260620 08:55:13.447296 8597 ts_manager.cc:194] Registered new tserver with Master: 2de537629ddf4daa96ccd0e89e077da3 (127.0.98.2:38035)
I20260620 08:55:13.448055 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:55:13.460625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45335
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:13.566895 8927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:13.567145 8927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:13.567189 8927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:13.571275 8927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:13.571336 8927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:13.571394 8927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:13.576588 8927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:13.577575 8927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:13.578511 8927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:13.587850 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.587894 8927 server_base.cc:1034] running on GCE node
W20260620 08:55:13.587896 8932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:13.588053 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.588503 8927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:13.589093 8927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:13.590323 8927 hybrid_clock.cc:648] HybridClock initialized: now 1781945713590294 us; error 38 us; skew 500 ppm
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:13.592960 8927 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:55:13.593755 8927 webserver.cc:460] Webserver started at http://127.0.98.3:35871/ using document root <none> and password file <none>
I20260620 08:55:13.594242 8927 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:13.594295 8927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:13.594447 8927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:13.596397 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "f391c42aa7294e79abc3147c3ea66116"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "4786220f7589d16ba70e616a29cbc3f2"
server_key_iv: "2a6d2aa6b7940bca53261aa52d6a69b0"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.596874 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f391c42aa7294e79abc3147c3ea66116"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "4786220f7589d16ba70e616a29cbc3f2"
server_key_iv: "2a6d2aa6b7940bca53261aa52d6a69b0"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.600401 8927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:13.602687 8942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.603904 8927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:13.604008 8927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "f391c42aa7294e79abc3147c3ea66116"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "4786220f7589d16ba70e616a29cbc3f2"
server_key_iv: "2a6d2aa6b7940bca53261aa52d6a69b0"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.604120 8927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:13.634087 8927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:13.637324 8927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:13.637508 8927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:13.638077 8927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:13.638855 8927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:13.638900 8927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.638952 8927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:13.638978 8927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.647202 8927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32975
I20260620 08:55:13.647257 9055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32975 every 8 connection(s)
I20260620 08:55:13.648075 8927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:13.656744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 8927
I20260620 08:55:13.656857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:13.657119 392 external_mini_cluster.cc:1428] Setting key 6dac08255fa3fb418d244b4003e1e9d8
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:13.661114 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:13.649799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45505 (local address 127.0.98.62:45335)
0620 08:55:13.649967 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:13.649971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:13.650731 (+ 760us) server_negotiation.cc:407] Connection header received
0620 08:55:13.652095 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:13.652099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:13.652143 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:13.652223 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:13.653372 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.653918 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.654957 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.655171 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.655250 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:13.657727 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:13.657731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:13.657734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:13.657760 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:13.659255 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:13.659751 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:13.659754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:13.659755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:13.659834 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:13.660175 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:13.660177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:13.660178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:13.660350 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:13.660422 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:13.660892 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 08:55:13.660997 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 08:55:13.661993 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:13.650149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:45505)
0620 08:55:13.650581 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:13.650621 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:13.651884 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:13.652412 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:13.652418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:13.652759 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:13.653234 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:13.653247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.654005 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.654007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:13.654860 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:13.654868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:13.655315 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:13.655318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:13.655457 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:13.656060 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:13.656081 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:13.657562 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:13.659356 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:13.659361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:13.659373 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:13.659660 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:13.659936 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:13.659939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:13.659941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:13.660096 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:13.660421 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:13.660426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:13.660648 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 08:55:13.660792 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 08:55:13.661029 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260620 08:55:13.663172 9056 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45335
I20260620 08:55:13.663378 9056 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:13.663885 9056 heartbeater.cc:511] Master 127.0.98.62:45335 requested a full tablet report, sending...
I20260620 08:55:13.664841 8597 ts_manager.cc:194] Registered new tserver with Master: f391c42aa7294e79abc3147c3ea66116 (127.0.98.3:32975)
I20260620 08:55:13.665408 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45505
I20260620 08:55:13.670737 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 08:55:13.742799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 8659
I20260620 08:55:13.747805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 8793
I20260620 08:55:13.752254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 8927
I20260620 08:55:13.756572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 8565
I20260620 08:55:13.761471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45335
--webserver_interface=127.0.98.62
--webserver_port=37101
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:13.865859 9065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:13.866120 9065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:13.866158 9065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:13.870110 9065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:13.870164 9065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:13.870182 9065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:13.870198 9065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:13.870213 9065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:13.875319 9065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:13.876297 9065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:13.877259 9065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:13.885138 9070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:13.885135 9071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.885290 9065 server_base.cc:1034] running on GCE node
W20260620 08:55:13.885198 9073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:13.885900 9065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:13.886976 9065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:13.888144 9065 hybrid_clock.cc:648] HybridClock initialized: now 1781945713888127 us; error 31 us; skew 500 ppm
Jun 20 08:55:13 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:13.890959 9065 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:55:13.891858 9065 webserver.cc:460] Webserver started at http://127.0.98.62:37101/ using document root <none> and password file <none>
I20260620 08:55:13.892364 9065 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:13.892441 9065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:13.896059 9065 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:13.898100 9080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:13.899245 9065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:13.899353 9065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "99cd76fe589041dca97c14dcd9f59e98"
format_stamp: "Formatted at 2026-06-20 08:55:12 on dist-test-slave-fb1v"
server_key: "8e91cc68187b1ed41999f6de606f5fd9"
server_key_iv: "fb8d402ea370172808a4d44975e4821d"
server_key_version: "encryptionkey@0"
I20260620 08:55:13.899896 9065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:13.910979 9065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:13.914023 9065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:13.914418 9065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:13.921078 9065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45335
I20260620 08:55:13.921094 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45335 every 8 connection(s)
I20260620 08:55:13.922080 9065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:13.926198 9133 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:55:13.926949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9065
I20260620 08:55:13.927345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36299
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45335
--webserver_port=35803
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:55:13.927850 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Bootstrap starting.
I20260620 08:55:13.943841 9133 log.cc:824] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:13.950006 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:55:13.950323 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Bootstrap complete.
I20260620 08:55:13.954130 9133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:13.954499 9133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cd76fe589041dca97c14dcd9f59e98, State: Initialized, Role: FOLLOWER
I20260620 08:55:13.954964 9133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:13.955080 9133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:13.955142 9133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:13.955227 9133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:55:13.957011 9133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:13.957301 9133 leader_election.cc:302] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99cd76fe589041dca97c14dcd9f59e98; no voters:
I20260620 08:55:13.957569 9133 leader_election.cc:288] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:55:13.957670 9137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:55:13.957849 9137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [term 2 LEADER]: Becoming Leader. State: Replica: 99cd76fe589041dca97c14dcd9f59e98, State: Running, Role: LEADER
I20260620 08:55:13.958129 9137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } }
I20260620 08:55:13.958717 9133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:13.958961 9137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99cd76fe589041dca97c14dcd9f59e98. Latest consensus state: current_term: 2 leader_uuid: "99cd76fe589041dca97c14dcd9f59e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } } }
I20260620 08:55:13.959075 9137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:13.959398 9141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:13.959478 9138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "99cd76fe589041dca97c14dcd9f59e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99cd76fe589041dca97c14dcd9f59e98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45335 } } }
I20260620 08:55:13.959679 9138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:13.962512 9141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:13.963151 9141 catalog_manager.cc:1259] Loaded cluster ID: 94499d82dddb46f088eb620cbda65f4b
I20260620 08:55:13.963200 9141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:13.964954 9141 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:13.965535 9141 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99cd76fe589041dca97c14dcd9f59e98: Loaded TSK: 0
I20260620 08:55:13.966496 9141 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:55:14.050418 9135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:14.050686 9135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:14.050747 9135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:14.055253 9135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:14.055328 9135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:14.055413 9135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:14.061120 9135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35803
--tserver_master_addrs=127.0.98.62:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:14.062130 9135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:14.063158 9135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:14.073338 9160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:14.073282 9159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:14.073385 9162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:14.073860 9135 server_base.cc:1034] running on GCE node
I20260620 08:55:14.074311 9135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:14.075100 9135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:14.076297 9135 hybrid_clock.cc:648] HybridClock initialized: now 1781945714076274 us; error 33 us; skew 500 ppm
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945714, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:14.079789 9135 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 08:55:14.080642 9135 webserver.cc:460] Webserver started at http://127.0.98.1:35803/ using document root <none> and password file <none>
I20260620 08:55:14.081207 9135 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:14.081295 9135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:14.085299 9135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:55:14.087424 9169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.088738 9135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:14.088845 9135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "67fee44869394924b15527033cb52049"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "d10b907eef5800c5358f7260038c1d2c"
server_key_iv: "72884c131b01a90414e33087b02032bc"
server_key_version: "encryptionkey@0"
I20260620 08:55:14.089847 9135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:14.102953 9135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:14.105944 9135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:14.106110 9135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:14.106649 9135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:14.107425 9135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:14.107478 9135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.107546 9135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:14.107573 9135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.115682 9135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36299
I20260620 08:55:14.115746 9282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36299 every 8 connection(s)
I20260620 08:55:14.116629 9135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:14.123720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9135
I20260620 08:55:14.124114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38035
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45335
--webserver_port=36959
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:55:14.129196 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.118625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42789 (local address 127.0.98.62:45335)
0620 08:55:14.119073 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.119093 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.119301 (+ 208us) server_negotiation.cc:407] Connection header received
0620 08:55:14.120777 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.120787 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.121063 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.121341 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.122425 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.123387 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.124589 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.125026 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.125178 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.128562 (+ 3384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42789: BlockingRecv error: recv got EOF from 127.0.98.1:42789 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":100,"threads_started":1}
I20260620 08:55:14.129281 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.118701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:42789)
0620 08:55:14.119158 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.119197 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.120397 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.121427 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.121435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.121833 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.122271 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.122283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.123510 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.123513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.124478 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.124489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.125171 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.125175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.125341 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.125911 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.125926 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.128351 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
W20260620 08:55:14.129695 9283 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45335 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45335: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:14.134717 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.130169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:56671)
0620 08:55:14.130282 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.130296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.130371 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.130596 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.130598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.130749 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.130832 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.130837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.131521 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.131524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.132092 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.132101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.132489 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.132493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.132576 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.133002 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.133013 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.134598 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 08:55:14.135022 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.130241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56671 (local address 127.0.98.62:45335)
0620 08:55:14.130332 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.130335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.130347 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:14.130451 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.130454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.130508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.130574 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.130928 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.131427 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.132195 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.132400 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.132482 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.134747 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56671: BlockingRecv error: recv got EOF from 127.0.98.1:56671 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:14.139848 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.135154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:54461)
0620 08:55:14.135261 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.135273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.135341 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.135707 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.135710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.135823 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.135898 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.135902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.136843 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.136845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.137372 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.137379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.137772 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.137776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.137844 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.138249 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.138263 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.139737 (+ 1474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:55:14.139936 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.135365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54461 (local address 127.0.98.62:45335)
0620 08:55:14.135482 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.135486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.135498 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:14.135530 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.135533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.135572 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.135689 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.135984 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.136750 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.137479 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.137683 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.137765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.139848 (+ 2083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54461: BlockingRecv error: recv got EOF from 127.0.98.1:54461 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 08:55:14.140108 9283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:55:14.234368 9287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:14.234633 9287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:14.234676 9287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:14.239257 9287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:14.239364 9287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:14.239440 9287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:14.245994 9287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36959
--tserver_master_addrs=127.0.98.62:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:14.247043 9287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:14.248049 9287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:14.257196 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:14.257148 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:14.257210 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:14.257273 9287 server_base.cc:1034] running on GCE node
I20260620 08:55:14.257777 9287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:14.258446 9287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:14.259652 9287 hybrid_clock.cc:648] HybridClock initialized: now 1781945714259634 us; error 27 us; skew 500 ppm
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945714, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:14.262388 9287 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 08:55:14.263200 9287 webserver.cc:460] Webserver started at http://127.0.98.2:36959/ using document root <none> and password file <none>
I20260620 08:55:14.263731 9287 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:14.263804 9287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:14.267467 9287 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:14.269594 9302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.271154 9287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:55:14.271265 9287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "2de537629ddf4daa96ccd0e89e077da3"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "665fb70112907411493e2b25150d39f0"
server_key_iv: "8d0c93aa3f15051e1236e78e34a76e17"
server_key_version: "encryptionkey@0"
I20260620 08:55:14.271831 9287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:14.300971 9287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:14.304227 9287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:14.304397 9287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:14.304977 9287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:14.305754 9287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:14.305799 9287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.305838 9287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:14.305864 9287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.314103 9287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38035
I20260620 08:55:14.314141 9415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38035 every 8 connection(s)
I20260620 08:55:14.315056 9287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:14.320736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9287
I20260620 08:55:14.321149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32975
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45335
--webserver_port=35871
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:55:14.327145 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.317018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59933 (local address 127.0.98.62:45335)
0620 08:55:14.317293 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.317297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.317973 (+ 676us) server_negotiation.cc:407] Connection header received
0620 08:55:14.319282 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.319285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.319353 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.319432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.320823 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.321444 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.322906 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.323139 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.323229 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.327021 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59933: BlockingRecv error: recv got EOF from 127.0.98.2:59933 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 08:55:14.327811 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.317278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:59933)
0620 08:55:14.317791 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.317837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.319071 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.319643 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.319651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.320009 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.320499 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.320517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.321581 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.321585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.322788 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.322798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.323280 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.323284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.323441 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.324264 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.324289 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.326889 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":131,"threads_started":1}
W20260620 08:55:14.328298 9416 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45335 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45335: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:14.334227 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.328790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:39147)
0620 08:55:14.328914 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.328928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.329069 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.329351 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.329353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.329502 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.329598 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.329604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.330589 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.330592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.331160 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.331168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.331788 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.331791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.331869 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.332393 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.332405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.334098 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:55:14.334301 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.328887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39147 (local address 127.0.98.62:45335)
0620 08:55:14.329020 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.329024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.329040 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:55:14.329195 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.329198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.329249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.329338 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.329700 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.330459 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.331297 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.331670 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.331783 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.334207 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39147: BlockingRecv error: recv got EOF from 127.0.98.2:39147 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:14.339136 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.334791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:51335)
0620 08:55:14.334893 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.334907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.334977 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.335210 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.335213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.335336 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.335400 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.335404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.336145 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.336147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.336651 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.336658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.337042 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.337045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.337115 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.337523 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.337534 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.339005 (+ 1471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:55:14.339344 9416 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:55:14.339448 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.334875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51335 (local address 127.0.98.62:45335)
0620 08:55:14.334969 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.334973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.334986 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:14.335055 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.335057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.335095 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.335153 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.335496 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.336032 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.336751 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.336953 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.337033 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.339127 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51335: BlockingRecv error: recv got EOF from 127.0.98.2:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 08:55:14.433151 9419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:14.433416 9419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:14.433477 9419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:14.438263 9419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:14.438328 9419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:14.438414 9419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:14.444072 9419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35871
--tserver_master_addrs=127.0.98.62:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:14.445220 9419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:14.446244 9419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:14.456226 9427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:14.456235 9424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:14.456223 9425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:14.456292 9419 server_base.cc:1034] running on GCE node
I20260620 08:55:14.456950 9419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:14.457595 9419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:14.458796 9419 hybrid_clock.cc:648] HybridClock initialized: now 1781945714458775 us; error 32 us; skew 500 ppm
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945714, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:14.461696 9419 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 08:55:14.462694 9419 webserver.cc:460] Webserver started at http://127.0.98.3:35871/ using document root <none> and password file <none>
I20260620 08:55:14.463320 9419 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:14.463419 9419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:14.467404 9419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:14.469930 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.471518 9419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:55:14.471679 9419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "f391c42aa7294e79abc3147c3ea66116"
format_stamp: "Formatted at 2026-06-20 08:55:13 on dist-test-slave-fb1v"
server_key: "4786220f7589d16ba70e616a29cbc3f2"
server_key_iv: "2a6d2aa6b7940bca53261aa52d6a69b0"
server_key_version: "encryptionkey@0"
I20260620 08:55:14.472225 9419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:14.493089 9419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:14.496136 9419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:14.496325 9419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:14.496938 9419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:14.497778 9419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:14.497830 9419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.497885 9419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:14.497913 9419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:14.507069 9419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32975
I20260620 08:55:14.507121 9547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32975 every 8 connection(s)
I20260620 08:55:14.508102 9419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:55:14.518296 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.509818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43359 (local address 127.0.98.62:45335)
0620 08:55:14.509959 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.509962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.510708 (+ 746us) server_negotiation.cc:407] Connection header received
0620 08:55:14.512009 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.512012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.512059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.512129 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.513186 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.513728 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.514810 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.515025 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.515108 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.518150 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43359: BlockingRecv error: recv got EOF from 127.0.98.3:43359 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:55:14.518340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9419
I20260620 08:55:14.518829 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.510146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:43359)
0620 08:55:14.510553 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.510603 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.511827 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.512266 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.512272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.512606 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.513038 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.513050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.513840 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.513842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.514693 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.514702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.515159 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.515162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.515292 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.515834 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.515853 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.517993 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
W20260620 08:55:14.519241 9548 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45335 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45335: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:14.524523 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.519641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:42219)
0620 08:55:14.519719 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.519731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.519795 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.520078 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.520080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.520203 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.520285 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.520289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.521077 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.521080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.521702 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.521710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.522105 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.522109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.522194 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.522598 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.522619 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.524410 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 08:55:14.524740 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.519708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42219 (local address 127.0.98.62:45335)
0620 08:55:14.519877 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.519880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.519890 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:55:14.519923 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.519926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.519964 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.520046 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.520381 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.520984 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.521802 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.522017 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.522098 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.524531 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42219: BlockingRecv error: recv got EOF from 127.0.98.3:42219 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260620 08:55:14.527393 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.519235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.0.98.62:45335)
0620 08:55:14.519357 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.519361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.519764 (+ 403us) server_negotiation.cc:407] Connection header received
0620 08:55:14.519802 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.519804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.519847 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.519910 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.520660 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.521239 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.522062 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.522258 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.522338 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.523668 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:14.523675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:14.523695 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:14.523734 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:14.525569 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:14.525946 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:14.525963 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:14.525968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:14.526074 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:14.526308 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:14.526311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:14.526313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:14.526635 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:14.526814 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:14.527090 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 08:55:14.527294 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:14 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:14.529374 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:14.525069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:46975)
0620 08:55:14.525176 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:14.525190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:14.525266 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:14.525454 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:14.525457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:14.525588 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:14.525652 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.525657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.526298 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.526301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.526798 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:14.526804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.527292 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.527295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:14.527350 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.527740 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:14.527753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:14.529273 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:55:14.529453 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:14.525117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46975 (local address 127.0.98.62:45335)
0620 08:55:14.525242 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:14.525245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:14.525257 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:14.525333 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:14.525336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:14.525372 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:14.525434 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:14.525746 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.526207 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.526894 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:14.527108 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:14.527188 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:14.529371 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46975: BlockingRecv error: recv got EOF from 127.0.98.3:46975 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 08:55:14.529635 9548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 08:55:15 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:15 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:15.145932 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:15.140641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:42557)
0620 08:55:15.140882 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:15.140896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:15.140982 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:15.141195 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:15.141197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:15.141365 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:15.141476 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:15.141482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.142406 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.142409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:15.143014 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:15.143023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.143454 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.143457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:15.143514 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:15.144046 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:15.144058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:15.145804 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
W20260620 08:55:15.146136 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:15.140746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42557 (local address 127.0.98.62:45335)
0620 08:55:15.140973 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:15.140976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:15.140989 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:15.141047 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:15.141049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:15.141091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:15.141170 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:15.141562 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.142303 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.143146 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.143369 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.143450 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:15.145973 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42557: BlockingRecv error: recv got EOF from 127.0.98.1:42557 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
Jun 20 08:55:15 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:15 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:15.345245 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:15.340013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:35113)
0620 08:55:15.340294 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:15.340308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:15.340422 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:15.340649 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:15.340651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:15.340825 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:15.340956 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:15.340962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.341685 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.341688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:15.342293 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:15.342301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.342755 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.342758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:15.342837 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:15.343323 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:15.343335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:15.345121 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
W20260620 08:55:15.345353 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:15.340150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35113 (local address 127.0.98.62:45335)
0620 08:55:15.340289 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:15.340293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:15.340370 (+ 77us) server_negotiation.cc:407] Connection header received
0620 08:55:15.340498 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:15.340500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:15.340544 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:15.340611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:15.341056 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.341570 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.342412 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.342628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.342735 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:15.345258 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35113: BlockingRecv error: recv got EOF from 127.0.98.2:35113 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 08:55:15 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:15 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:15.535427 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:15.530219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:56545)
0620 08:55:15.530455 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:15.530469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:15.530566 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:15.530827 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:15.530829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:15.530997 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:15.531118 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:15.531124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.531901 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.531904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:15.532485 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:15.532493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.532912 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.532916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:15.533003 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:15.533511 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:15.533524 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:15.535305 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
W20260620 08:55:15.535523 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:15.530316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56545 (local address 127.0.98.62:45335)
0620 08:55:15.530458 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:15.530461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:15.530526 (+ 65us) server_negotiation.cc:407] Connection header received
0620 08:55:15.530654 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:15.530656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:15.530709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:15.530767 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:15.531228 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.531776 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.532576 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:15.532808 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:15.532873 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:15.535429 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56545: BlockingRecv error: recv got EOF from 127.0.98.3:56545 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 08:55:16 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:16 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:16.152289 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:16.146706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:55941)
0620 08:55:16.146970 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:16.146985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:16.147080 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:16.147317 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:16.147319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:16.147483 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:16.147582 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:16.147587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.148339 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.148342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:16.148927 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:16.148936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.149445 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.149448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:16.149514 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:16.149985 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:16.149997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:16.152149 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
W20260620 08:55:16.152426 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:16.146809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55941 (local address 127.0.98.62:45335)
0620 08:55:16.147039 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:16.147043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:16.147056 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:16.147165 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:16.147167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:16.147221 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:16.147294 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:16.147693 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.148215 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.149053 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.149323 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.149401 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:16.152299 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55941: BlockingRecv error: recv got EOF from 127.0.98.1:55941 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":57,"threads_started":1}
Jun 20 08:55:16 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:16 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:16.353246 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:16.346227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:41045)
0620 08:55:16.346529 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:16.346544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:16.346683 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:16.346931 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:16.346934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:16.347114 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:16.347221 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:16.347227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.348312 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.348315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:16.349128 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:16.349140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.350146 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.350156 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:16.350248 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:16.350735 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:16.350747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:16.353082 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":90,"threads_started":1}
W20260620 08:55:16.353320 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:16.346316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41045 (local address 127.0.98.62:45335)
0620 08:55:16.346478 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:16.346483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:16.346630 (+ 147us) server_negotiation.cc:407] Connection header received
0620 08:55:16.346779 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:16.346782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:16.346828 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:16.346892 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:16.347413 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.348177 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.349272 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.349571 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.349684 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:16.353223 (+ 3539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41045: BlockingRecv error: recv got EOF from 127.0.98.2:41045 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 08:55:16 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:16 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:16.541672 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:16.536120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:56127)
0620 08:55:16.536373 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:16.536387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:16.536472 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:16.536702 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:16.536704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:16.536880 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:16.536985 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:16.537004 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.537776 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.537779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:16.538355 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:16.538363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.539032 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.539035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:16.539105 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:16.539652 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:16.539665 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:16.541479 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
W20260620 08:55:16.541752 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:16.536339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56127 (local address 127.0.98.62:45335)
0620 08:55:16.536468 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:16.536471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:16.536485 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:55:16.536563 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:16.536565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:16.536612 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:16.536681 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:16.537132 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.537651 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.538533 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:16.538875 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:16.538996 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:16.541649 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56127: BlockingRecv error: recv got EOF from 127.0.98.3:56127 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 08:55:17 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:17 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:17.158454 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:17.153009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:56327)
0620 08:55:17.153310 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:17.153322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:17.153411 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:17.153648 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:17.153650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:17.153815 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:17.153907 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:17.153912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.154699 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.154703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:17.155415 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:17.155428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.155988 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.155993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:17.156085 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:17.156553 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:17.156564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:17.158306 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
W20260620 08:55:17.158583 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:17.153143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56327 (local address 127.0.98.62:45335)
0620 08:55:17.153360 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:17.153363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:17.153378 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:55:17.153499 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:17.153501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:17.153543 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:17.153627 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:17.154018 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.154556 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.155546 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.155871 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.155975 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:17.158449 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56327: BlockingRecv error: recv got EOF from 127.0.98.1:56327 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
Jun 20 08:55:17 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:17 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:17.359493 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:17.354069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:47563)
0620 08:55:17.354374 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:17.354391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:17.354496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:17.354794 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:17.354797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:17.355001 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:17.355091 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:17.355104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.356007 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.356011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:17.356572 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:17.356580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.356988 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.356991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:17.357071 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:17.357531 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:17.357543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:17.359349 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
W20260620 08:55:17.359567 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:17.354445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47563 (local address 127.0.98.62:45335)
0620 08:55:17.354573 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:17.354577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:17.354589 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:55:17.354630 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:17.354633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:17.354677 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:17.354747 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:17.355203 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.355905 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.356674 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.356896 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.356979 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:17.359474 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47563: BlockingRecv error: recv got EOF from 127.0.98.2:47563 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 08:55:17 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:17 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:17.548473 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:17.542462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:34867)
0620 08:55:17.542725 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:17.542738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:17.542834 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:17.543045 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:17.543047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:17.543231 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:17.543322 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:17.543327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.544410 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.544413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:17.545013 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:17.545022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.545504 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.545507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:17.545565 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:17.546044 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:17.546056 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:17.548324 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
W20260620 08:55:17.548570 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:17.542549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34867 (local address 127.0.98.62:45335)
0620 08:55:17.542683 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:17.542686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:17.542801 (+ 115us) server_negotiation.cc:407] Connection header received
0620 08:55:17.542902 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:17.542904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:17.542950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:17.543018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:17.543540 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.544252 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.545153 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:17.545413 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:17.545505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:17.548474 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34867: BlockingRecv error: recv got EOF from 127.0.98.3:34867 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 08:55:18 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:18 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:18.165940 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:18.159257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:49805)
0620 08:55:18.159567 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:18.159581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:18.159720 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:18.160001 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:18.160004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:18.160181 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:18.160275 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:18.160280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.161188 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.161191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:18.161762 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:18.161771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.162417 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.162420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:18.162492 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:18.163324 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:18.163345 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:18.165807 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
W20260620 08:55:18.166051 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:18.159302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49805 (local address 127.0.98.62:45335)
0620 08:55:18.159573 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:18.159577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:18.159675 (+ 98us) server_negotiation.cc:407] Connection header received
0620 08:55:18.159820 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:18.159824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:18.159875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:18.159944 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:18.160411 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.160983 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.161994 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.162230 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.162321 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:18.165943 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49805: BlockingRecv error: recv got EOF from 127.0.98.1:49805 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
Jun 20 08:55:18 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:18 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:18.365355 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:18.360380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:49859)
0620 08:55:18.360637 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:18.360653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:18.360767 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:18.361023 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:18.361026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:18.361193 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:18.361288 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:18.361293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.361986 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.361988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:18.362523 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:18.362530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.362905 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.362907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:18.362966 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:18.363459 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:18.363473 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:18.365243 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
W20260620 08:55:18.365473 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:18.360405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49859 (local address 127.0.98.62:45335)
0620 08:55:18.360552 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:18.360557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:18.360720 (+ 163us) server_negotiation.cc:407] Connection header received
0620 08:55:18.360860 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:18.360863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:18.360917 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:18.360986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:18.361387 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.361907 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.362603 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.362824 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.362896 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:18.365372 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49859: BlockingRecv error: recv got EOF from 127.0.98.2:49859 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 08:55:18 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:18 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:18.559203 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:18.549278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:36493)
0620 08:55:18.549508 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:18.549522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:18.549615 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:18.549838 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:18.549840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:18.550005 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:18.550097 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:18.550101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.550801 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.550804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:18.551389 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:18.551397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.551875 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.551879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:18.551945 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:18.556511 (+ 4566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:18.556525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:18.559043 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
W20260620 08:55:18.559314 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:18.549398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36493 (local address 127.0.98.62:45335)
0620 08:55:18.549510 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:18.549513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:18.549579 (+ 66us) server_negotiation.cc:407] Connection header received
0620 08:55:18.549698 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:18.549700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:18.549744 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:18.549812 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:18.550192 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.550688 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.551505 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:18.551745 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:18.551816 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:18.559216 (+ 7400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36493: BlockingRecv error: recv got EOF from 127.0.98.3:36493 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:55:19 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:19 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:19.171873 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:19.166718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:52751)
0620 08:55:19.166970 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:19.166984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:19.167067 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:19.167296 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:19.167298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:19.167466 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:19.167558 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:19.167563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.168336 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.168339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:19.168920 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:19.168929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.169398 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.169400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:19.169462 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:19.169936 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:19.169947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:19.171755 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
W20260620 08:55:19.171998 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:19.166801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52751 (local address 127.0.98.62:45335)
0620 08:55:19.167058 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:19.167061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:19.167074 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:19.167148 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:19.167150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:19.167191 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:19.167264 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:19.167682 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.168232 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.169037 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.169292 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.169380 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:19.171884 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52751: BlockingRecv error: recv got EOF from 127.0.98.1:52751 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
Jun 20 08:55:19 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:19 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:19.371030 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:19.366092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:60967)
0620 08:55:19.366347 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:19.366360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:19.366447 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:19.366657 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:19.366659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:19.366825 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:19.366912 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:19.366917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.367708 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.367717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:19.368294 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:19.368303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.368725 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.368728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:19.368802 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:19.369246 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:19.369258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:19.370874 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
W20260620 08:55:19.371157 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:19.366249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60967 (local address 127.0.98.62:45335)
0620 08:55:19.366361 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:19.366365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:19.366406 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:55:19.366516 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:19.366518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:19.366558 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:19.366634 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:19.367030 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.367562 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.368401 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.368623 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.368700 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:19.371037 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60967: BlockingRecv error: recv got EOF from 127.0.98.2:60967 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 08:55:19 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:19 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:19.564791 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:19.560076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:42701)
0620 08:55:19.560333 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:19.560347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:19.560438 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:19.560637 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:19.560639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:19.560812 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:19.560919 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:19.560926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.561571 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.561573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:19.562126 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:19.562132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.562519 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.562521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:19.562583 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:19.563021 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:19.563031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:19.564667 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
W20260620 08:55:19.564890 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:19.560176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42701 (local address 127.0.98.62:45335)
0620 08:55:19.560300 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:19.560304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:19.560430 (+ 126us) server_negotiation.cc:407] Connection header received
0620 08:55:19.560504 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:19.560506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:19.560549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:19.560616 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:19.561001 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.561495 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.562227 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:19.562434 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:19.562518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:19.564784 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42701: BlockingRecv error: recv got EOF from 127.0.98.3:42701 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:55:20 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:20 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:20.178519 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:20.172777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:59323)
0620 08:55:20.173023 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:20.173037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:20.173126 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:20.173404 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:20.173408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:20.173612 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:20.173728 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:20.173736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.174511 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.174514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:20.175194 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:20.175226 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.175729 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.175732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:20.175813 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:20.176305 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:20.176316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:20.178374 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
W20260620 08:55:20.178625 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:20.172872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59323 (local address 127.0.98.62:45335)
0620 08:55:20.173164 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:20.173168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:20.173183 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:55:20.173224 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:20.173227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:20.173278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:20.173377 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:20.173837 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.174410 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.175357 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.175597 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.175705 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:20.178513 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59323: BlockingRecv error: recv got EOF from 127.0.98.1:59323 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
Jun 20 08:55:20 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:20 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:20.377211 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:20.371974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:58817)
0620 08:55:20.372214 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:20.372227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:20.372328 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:20.372547 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:20.372549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:20.372711 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:20.372800 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:20.372805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.373654 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.373658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:20.374289 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:20.374300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.374721 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.374725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:20.374789 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:20.375234 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:20.375262 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:20.377043 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
W20260620 08:55:20.377313 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:20.372070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58817 (local address 127.0.98.62:45335)
0620 08:55:20.372185 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:20.372189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:20.372284 (+ 95us) server_negotiation.cc:407] Connection header received
0620 08:55:20.372416 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:20.372418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:20.372461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:20.372517 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:20.372917 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.373569 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.374387 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.374608 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.374686 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:20.377215 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58817: BlockingRecv error: recv got EOF from 127.0.98.2:58817 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 08:55:20 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:20 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:20.570577 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:20.565456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:53975)
0620 08:55:20.565709 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:20.565722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:20.565818 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:20.566066 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:20.566069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:20.566244 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:20.566338 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:20.566343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.567060 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.567063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:20.567665 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:20.567674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.568091 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.568094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:20.568157 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:20.568592 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:20.568602 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:20.570447 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
W20260620 08:55:20.570709 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:20.565546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53975 (local address 127.0.98.62:45335)
0620 08:55:20.565659 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:20.565662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:20.565780 (+ 118us) server_negotiation.cc:407] Connection header received
0620 08:55:20.565882 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:20.565884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:20.565928 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:20.565997 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:20.566438 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.566947 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.567773 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:20.567982 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:20.568063 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:20.570600 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53975: BlockingRecv error: recv got EOF from 127.0.98.3:53975 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 08:55:21 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:21 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:21.184684 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:21.179199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:40679)
0620 08:55:21.179484 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:21.179497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:21.179596 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:21.179934 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:21.179937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:21.180105 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:21.180203 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:21.180209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.181066 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.181069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:21.181632 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:21.181642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.182148 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.182151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:21.182224 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:21.182859 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:21.182871 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:21.184568 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":67,"threads_started":1}
W20260620 08:55:21.184966 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:21.179303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40679 (local address 127.0.98.62:45335)
0620 08:55:21.179506 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:21.179509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:21.179547 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:55:21.179736 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:21.179739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:21.179791 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:21.179873 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:21.180426 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.180950 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.181759 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.181986 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.182115 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:21.184719 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40679: BlockingRecv error: recv got EOF from 127.0.98.1:40679 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
Jun 20 08:55:21 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:21 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:21.383142 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:21.378083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:41131)
0620 08:55:21.378366 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:21.378379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:21.378484 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:21.378739 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:21.378742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:21.378920 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:21.379013 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:21.379018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.379805 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.379808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:21.380373 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:21.380381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.380809 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.380812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:21.380869 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:21.381328 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:21.381339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:21.383028 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":70,"threads_started":1}
W20260620 08:55:21.383329 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:21.378194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41131 (local address 127.0.98.62:45335)
0620 08:55:21.378341 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:21.378346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:21.378504 (+ 158us) server_negotiation.cc:407] Connection header received
0620 08:55:21.378546 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:21.378548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:21.378592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:21.378670 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:21.379131 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.379683 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.380488 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.380723 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.380802 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:21.383199 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41131: BlockingRecv error: recv got EOF from 127.0.98.2:41131 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 08:55:21 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:21 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:21.576967 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:21.571639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:56817)
0620 08:55:21.571920 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:21.571933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:21.572028 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:21.572248 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:21.572250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:21.572415 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:21.572507 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:21.572512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.573272 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.573275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:21.573857 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:21.573866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.574354 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.574357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:21.574423 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:21.574931 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:21.574943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:21.576827 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
W20260620 08:55:21.577091 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:21.571694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56817 (local address 127.0.98.62:45335)
0620 08:55:21.571807 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:21.571811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:21.572017 (+ 206us) server_negotiation.cc:407] Connection header received
0620 08:55:21.572107 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:21.572110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:21.572164 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:21.572224 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:21.572622 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.573153 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.573975 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:21.574215 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:21.574315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:21.576962 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56817: BlockingRecv error: recv got EOF from 127.0.98.3:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 08:55:22 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:22 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:22.190690 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:22.185466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:50059)
0620 08:55:22.185731 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:22.185743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:22.185841 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:22.186110 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:22.186112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:22.186279 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:22.186374 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:22.186379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.187092 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.187095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:22.187689 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:22.187698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.188099 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.188102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:22.188220 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:22.188711 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:22.188726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:22.190576 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
W20260620 08:55:22.190855 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:22.185562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50059 (local address 127.0.98.62:45335)
0620 08:55:22.185788 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:22.185792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:22.185806 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:55:22.185951 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:22.185954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:22.186008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:22.186091 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:22.186463 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.186994 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.187769 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.187977 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.188048 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:22.190732 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50059: BlockingRecv error: recv got EOF from 127.0.98.1:50059 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
Jun 20 08:55:22 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:22 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:22.389480 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:22.383868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:38587)
0620 08:55:22.384109 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:22.384121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:22.384214 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:22.384426 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:22.384428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:22.384593 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:22.384693 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:22.384698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.385691 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.385694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:22.386251 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:22.386259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.386758 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.386762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:22.386832 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:22.387358 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:22.387370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:22.389360 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
W20260620 08:55:22.389644 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:22.383940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38587 (local address 127.0.98.62:45335)
0620 08:55:22.384073 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:22.384076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:22.384207 (+ 131us) server_negotiation.cc:407] Connection header received
0620 08:55:22.384294 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:22.384296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:22.384341 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:22.384399 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:22.384806 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.385561 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.386370 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.386618 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.386712 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:22.389522 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38587: BlockingRecv error: recv got EOF from 127.0.98.2:38587 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 08:55:22 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:22 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:22.582686 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:22.577805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:51277)
0620 08:55:22.578044 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:22.578057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:22.578152 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:22.578342 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:22.578343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:22.578507 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:22.578599 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:22.578604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.579290 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.579293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:22.579895 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:22.579905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.580292 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.580294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:22.580348 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:22.580776 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:22.580786 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:22.582521 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
W20260620 08:55:22.582785 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:22.577866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51277 (local address 127.0.98.62:45335)
0620 08:55:22.577974 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:22.577978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:22.578106 (+ 128us) server_negotiation.cc:407] Connection header received
0620 08:55:22.578215 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:22.578217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:22.578261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:22.578321 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:22.578688 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.579183 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.579990 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:22.580208 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:22.580284 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:22.582664 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51277: BlockingRecv error: recv got EOF from 127.0.98.3:51277 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 08:55:23 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:23 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:23.198412 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:23.191735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:50937)
0620 08:55:23.192017 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:23.192030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:23.192144 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:23.192479 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:23.192482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:23.192669 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:23.192765 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:23.192771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.193655 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.193661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:23.194561 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:23.194588 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.195205 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.195209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:23.195322 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:23.196048 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:23.196064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:23.198295 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
W20260620 08:55:23.198578 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:23.191855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50937 (local address 127.0.98.62:45335)
0620 08:55:23.192074 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:23.192077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:23.192095 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:55:23.192286 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:23.192289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:23.192338 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:23.192425 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:23.192885 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.193446 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.194757 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.195054 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.195149 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:23.198443 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50937: BlockingRecv error: recv got EOF from 127.0.98.1:50937 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
Jun 20 08:55:23 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:23 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:23.396745 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:23.390511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:50743)
0620 08:55:23.391029 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:23.391051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:23.391223 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:23.391600 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:23.391631 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:23.391850 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:23.391948 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:23.391955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.392685 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.392688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:23.393342 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:23.393352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.393814 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.393823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:23.393909 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:23.394516 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:23.394528 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:23.396607 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":128,"threads_started":1}
W20260620 08:55:23.396893 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:23.390656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50743 (local address 127.0.98.62:45335)
0620 08:55:23.390987 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:23.390992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:23.391160 (+ 168us) server_negotiation.cc:407] Connection header received
0620 08:55:23.391413 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:23.391417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:23.391474 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:23.391559 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:23.392073 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.392584 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.393449 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.393657 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.393748 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:23.396773 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50743: BlockingRecv error: recv got EOF from 127.0.98.2:50743 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
Jun 20 08:55:23 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:23 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:23.591252 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:23.583476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:34449)
0620 08:55:23.583734 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:23.583747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:23.583840 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:23.584044 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:23.584046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:23.584235 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:23.584326 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:23.584331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.585011 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.585014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:23.585576 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:23.585584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.586001 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.586004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:23.586071 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:23.589405 (+ 3334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:23.589417 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:23.591134 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260620 08:55:23.591388 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:23.583528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34449 (local address 127.0.98.62:45335)
0620 08:55:23.583672 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:23.583676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:23.583806 (+ 130us) server_negotiation.cc:407] Connection header received
0620 08:55:23.583915 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:23.583917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:23.583961 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:23.584022 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:23.584408 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.584925 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.585678 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:23.585885 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:23.585956 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:23.591272 (+ 5316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34449: BlockingRecv error: recv got EOF from 127.0.98.3:34449 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 08:55:24 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:24 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:24.204352 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:24.199204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:38387)
0620 08:55:24.199471 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:24.199485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:24.199575 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:24.199877 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:24.199884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:24.200055 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:24.200153 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:24.200158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.200865 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.200868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:24.201432 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:24.201440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.201832 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.201834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:24.201887 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:24.202352 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:24.202363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:24.204240 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
W20260620 08:55:24.204502 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:24.199288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38387 (local address 127.0.98.62:45335)
0620 08:55:24.199497 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:24.199501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:24.199552 (+ 51us) server_negotiation.cc:407] Connection header received
0620 08:55:24.199681 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:24.199683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:24.199724 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:24.199799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:24.200257 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.200782 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.201534 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.201750 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.201825 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:24.204380 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38387: BlockingRecv error: recv got EOF from 127.0.98.1:38387 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":57,"threads_started":1}
Jun 20 08:55:24 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:24 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:24.402781 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:24.397558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:35253)
0620 08:55:24.397801 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:24.397814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:24.397911 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:24.398140 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:24.398142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:24.398310 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:24.398402 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:24.398407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.399195 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.399198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:24.399947 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:24.399957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.400446 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.400449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:24.400512 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:24.400947 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:24.400957 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:24.402650 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
W20260620 08:55:24.403038 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:24.397671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35253 (local address 127.0.98.62:45335)
0620 08:55:24.397825 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:24.397828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:24.397929 (+ 101us) server_negotiation.cc:407] Connection header received
0620 08:55:24.397981 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:24.397984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:24.398027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:24.398094 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:24.398521 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.399061 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.400052 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.400329 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.400417 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:24.402813 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35253: BlockingRecv error: recv got EOF from 127.0.98.2:35253 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 08:55:24 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:24 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:24.597139 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:24.592046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:43107)
0620 08:55:24.592305 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:24.592318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:24.592400 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:24.592610 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:24.592612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:24.592790 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:24.592897 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:24.592902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.593647 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.593649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:24.594249 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:24.594258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.594711 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.594714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:24.594776 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:24.595231 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:24.595242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:24.597015 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
W20260620 08:55:24.597275 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:24.592203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43107 (local address 127.0.98.62:45335)
0620 08:55:24.592346 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:24.592350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:24.592363 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:24.592475 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:24.592477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:24.592519 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:24.592587 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:24.593012 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.593567 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.594378 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:24.594605 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:24.594695 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:24.597159 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43107: BlockingRecv error: recv got EOF from 127.0.98.3:43107 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 08:55:25 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:25 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:25.210230 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:25.205065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:57215)
0620 08:55:25.205426 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:25.205440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:25.205523 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:25.205734 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:25.205736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:25.205916 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:25.206005 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:25.206011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.206708 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.206710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:25.207255 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:25.207262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.207781 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.207785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:25.207867 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:25.208436 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:25.208447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:25.210099 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":95,"threads_started":1}
W20260620 08:55:25.210367 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:25.205245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57215 (local address 127.0.98.62:45335)
0620 08:55:25.205523 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:25.205526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:25.205540 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:55:25.205579 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:25.205581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:25.205623 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:25.205703 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:25.206092 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.206632 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.207365 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.207668 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.207760 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:25.210235 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57215: BlockingRecv error: recv got EOF from 127.0.98.1:57215 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":70,"threads_started":1}
Jun 20 08:55:25 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:25 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:25.409077 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:25.403594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:60617)
0620 08:55:25.403854 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:25.403867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:25.403983 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:25.404303 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:25.404306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:25.404502 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:25.404617 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:25.404624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.405571 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.405574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:25.406141 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:25.406150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.406626 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.406629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:25.406697 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:25.407216 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:25.407229 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:25.408950 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
W20260620 08:55:25.409183 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:25.403701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60617 (local address 127.0.98.62:45335)
0620 08:55:25.403828 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:25.403832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:25.403995 (+ 163us) server_negotiation.cc:407] Connection header received
0620 08:55:25.404099 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:25.404101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:25.404172 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:25.404253 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:25.404750 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.405447 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.406271 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.406502 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.406610 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:25.409088 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60617: BlockingRecv error: recv got EOF from 127.0.98.2:60617 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:55:25 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:25 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:25.603073 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:25.598018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:47845)
0620 08:55:25.598266 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:25.598281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:25.598375 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:25.598589 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:25.598591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:25.598760 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:25.598864 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:25.598870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.599579 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.599581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:25.600177 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:25.600187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.600574 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.600576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:25.600634 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:25.601136 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:25.601148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:25.602947 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
W20260620 08:55:25.603158 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:25.598115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47845 (local address 127.0.98.62:45335)
0620 08:55:25.598246 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:25.598249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:25.598342 (+ 93us) server_negotiation.cc:407] Connection header received
0620 08:55:25.598444 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:25.598446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:25.598492 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:25.598560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:25.598964 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.599497 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.600282 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:25.600489 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:25.600564 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:25.603064 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47845: BlockingRecv error: recv got EOF from 127.0.98.3:47845 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:55:26 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:26 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:26.216156 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:26.210968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:41599)
0620 08:55:26.211241 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:26.211255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:26.211347 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:26.211553 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:26.211555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:26.211745 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:26.211848 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:26.211859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.212666 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.212669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:26.213255 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:26.213263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.213687 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.213690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:26.213749 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:26.214269 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:26.214280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:26.216042 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
W20260620 08:55:26.216288 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:26.211125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41599 (local address 127.0.98.62:45335)
0620 08:55:26.211333 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:26.211337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:26.211350 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:26.211415 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:26.211418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:26.211458 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:26.211530 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:26.211998 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.212540 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.213369 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.213593 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.213675 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:26.216161 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41599: BlockingRecv error: recv got EOF from 127.0.98.1:41599 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
Jun 20 08:55:26 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:26 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:26.415392 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:26.409871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:39387)
0620 08:55:26.410177 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:26.410192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:26.410331 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:26.410564 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:26.410567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:26.410765 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:26.410853 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:26.410859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.411643 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.411646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:26.412267 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:26.412276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.412791 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.412795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:26.412924 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:26.413444 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:26.413460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:26.415274 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":85,"threads_started":1}
W20260620 08:55:26.415532 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:26.409993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39387 (local address 127.0.98.62:45335)
0620 08:55:26.410161 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:26.410165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:26.410288 (+ 123us) server_negotiation.cc:407] Connection header received
0620 08:55:26.410419 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:26.410422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:26.410468 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:26.410533 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:26.410959 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.411504 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.412391 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.412680 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.412774 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:26.415416 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39387: BlockingRecv error: recv got EOF from 127.0.98.2:39387 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 08:55:26 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:26 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:26.609598 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:26.603904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:33801)
0620 08:55:26.604296 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:26.604314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:26.604504 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:26.604831 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:26.604835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:26.605017 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:26.605118 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:26.605124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.605947 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.605952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:26.606605 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:26.606620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.607092 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.607095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:26.607161 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:26.607701 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:26.607715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:26.609449 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":73,"threads_started":1}
W20260620 08:55:26.609679 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:26.604090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33801 (local address 127.0.98.62:45335)
0620 08:55:26.604247 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:26.604251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:26.604470 (+ 219us) server_negotiation.cc:407] Connection header received
0620 08:55:26.604598 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:26.604601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:26.604653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:26.604732 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:26.605249 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.605776 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.606738 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:26.606978 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:26.607062 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:26.609583 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33801: BlockingRecv error: recv got EOF from 127.0.98.3:33801 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 08:55:27 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:27 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:27.222070 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:27.216867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:52913)
0620 08:55:27.217125 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:27.217137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:27.217262 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:27.217488 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:27.217490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:27.217654 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:27.217748 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:27.217753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.218531 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.218534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:27.219193 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:27.219202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.219696 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.219700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:27.219787 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:27.220269 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:27.220280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:27.221960 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
W20260620 08:55:27.222208 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:27.216932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52913 (local address 127.0.98.62:45335)
0620 08:55:27.217138 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:27.217141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:27.217188 (+ 47us) server_negotiation.cc:407] Connection header received
0620 08:55:27.217335 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:27.217337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:27.217393 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:27.217468 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:27.217835 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.218445 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.219290 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.219528 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.219661 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:27.222079 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52913: BlockingRecv error: recv got EOF from 127.0.98.1:52913 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":52,"threads_started":1}
Jun 20 08:55:27 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:27 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:27.421146 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:27.416075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:34761)
0620 08:55:27.416351 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:27.416365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:27.416463 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:27.416648 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:27.416650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:27.416819 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:27.416907 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:27.416912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.417652 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.417655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:27.418211 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:27.418218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.418661 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.418665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:27.418732 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:27.419189 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:27.419200 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:27.421027 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
W20260620 08:55:27.421386 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:27.416215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34761 (local address 127.0.98.62:45335)
0620 08:55:27.416354 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:27.416358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:27.416426 (+ 68us) server_negotiation.cc:407] Connection header received
0620 08:55:27.416530 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:27.416532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:27.416574 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:27.416637 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:27.417028 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.417571 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.418322 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.418541 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.418627 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:27.421197 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34761: BlockingRecv error: recv got EOF from 127.0.98.2:34761 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 08:55:27 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:27 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:27.616180 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:27.610502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:45375)
0620 08:55:27.610786 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:27.610799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:27.610906 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:27.611193 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:27.611195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:27.611386 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:27.611477 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:27.611483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.612355 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.612359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:27.613095 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:27.613104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.613525 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.613527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:27.613583 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:27.614040 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:27.614051 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:27.616068 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
W20260620 08:55:27.616329 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:27.610593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45375 (local address 127.0.98.62:45335)
0620 08:55:27.610737 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:27.610741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:27.610886 (+ 145us) server_negotiation.cc:407] Connection header received
0620 08:55:27.611012 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:27.611015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:27.611064 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:27.611134 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:27.611634 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.612267 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.613212 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:27.613437 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:27.613519 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:27.616208 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45375: BlockingRecv error: recv got EOF from 127.0.98.3:45375 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 08:55:28 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:28 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:28.228363 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:28.222899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:39719)
0620 08:55:28.223174 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:28.223187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:28.223282 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:28.223559 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:28.223562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:28.223748 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:28.223850 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:28.223855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.224675 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.224678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:28.225245 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:28.225254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.225700 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.225702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:28.225763 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:28.226254 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:28.226265 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:28.228238 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
W20260620 08:55:28.228499 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:28.222936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39719 (local address 127.0.98.62:45335)
0620 08:55:28.223232 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:28.223236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:28.223249 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:55:28.223374 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:28.223377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:28.223431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:28.223515 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:28.223977 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.224553 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.225368 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.225588 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.225668 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:28.228370 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39719: BlockingRecv error: recv got EOF from 127.0.98.1:39719 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":70,"threads_started":1}
Jun 20 08:55:28 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:28 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:28.427361 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:28.421981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:53653)
0620 08:55:28.422273 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:28.422288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:28.422399 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:28.422667 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:28.422670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:28.422904 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:28.423002 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:28.423009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.423827 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.423831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:28.424378 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:28.424386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.424794 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.424798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:28.424873 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:28.425385 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:28.425397 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:28.427233 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
W20260620 08:55:28.427518 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:28.422076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53653 (local address 127.0.98.62:45335)
0620 08:55:28.422240 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:28.422244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:28.422357 (+ 113us) server_negotiation.cc:407] Connection header received
0620 08:55:28.422492 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:28.422495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:28.422545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:28.422632 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:28.423124 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.423721 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.424485 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.424700 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.424783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:28.427401 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53653: BlockingRecv error: recv got EOF from 127.0.98.2:53653 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 08:55:28 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:28 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:28.625510 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:28.616958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.3:59163)
0620 08:55:28.617204 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:28.617218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:28.617301 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:28.617557 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:28.617560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:28.617728 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:28.617822 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:28.617827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.618632 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.618636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:28.619432 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:28.619444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.620031 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.620035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:28.620119 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:28.623497 (+ 3378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:28.623510 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:28.625345 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20260620 08:55:28.625587 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:28.617118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59163 (local address 127.0.98.62:45335)
0620 08:55:28.617236 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:28.617240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:28.617256 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:55:28.617378 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:28.617380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:28.617422 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:28.617494 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:28.617969 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.618497 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.619579 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:28.619914 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:28.620018 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:28.625487 (+ 5469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59163: BlockingRecv error: recv got EOF from 127.0.98.3:59163 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 08:55:29 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:29 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:29.234651 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:29.229143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.1:40119)
0620 08:55:29.229413 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:29.229427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:29.229509 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:29.229780 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:29.229783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:29.229961 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:29.230051 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:29.230056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.230813 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.230817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:29.231376 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:29.231384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.231883 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.231886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:29.231952 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.232449 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:29.232460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:29.234526 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
W20260620 08:55:29.234803 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:29.229254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40119 (local address 127.0.98.62:45335)
0620 08:55:29.229447 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:29.229450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:29.229466 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:55:29.229615 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:29.229618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:29.229672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:29.229747 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:29.230160 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.230704 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.231490 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.231771 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.231849 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.234663 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40119: BlockingRecv error: recv got EOF from 127.0.98.1:40119 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
Jun 20 08:55:29 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:55:29 dist-test-slave-fb1v krb5kdc[8552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:55:29.433415 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:29.428184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45335 (local address 127.0.98.2:56443)
0620 08:55:29.428435 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:29.428449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:29.428551 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:29.428786 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:29.428788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:29.429045 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:29.429146 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:29.429152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.430016 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.430019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:29.430565 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:29.430573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.430997 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.431000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:29.431077 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.431556 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:29.431567 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:29.433268 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45335: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
W20260620 08:55:29.433506 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:29.428291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56443 (local address 127.0.98.62:45335)
0620 08:55:29.428438 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:29.428442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:29.428515 (+ 73us) server_negotiation.cc:407] Connection header received
0620 08:55:29.428632 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:29.428635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:29.428692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:29.428756 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:29.429350 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.429911 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.430679 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.430902 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.430985 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.433406 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56443: BlockingRecv error: recv got EOF from 127.0.98.2:56443 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:55:29.518633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9135
I20260620 08:55:29.523578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9287
I20260620 08:55:29.528057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9419
I20260620 08:55:29.532697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9065
2026-06-20T08:55:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17888 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T08:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:29Z Disabled control of system clock
I20260620 08:55:29.560937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38865
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 08:55:29.668365 9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:29.668625 9615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:29.668658 9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:29.672626 9615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:29.672683 9615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:29.672696 9615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:29.672712 9615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:29.672726 9615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:29.677393 9615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:29.678387 9615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:29.679683 9615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:29.688328 9621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:29.688364 9620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:29.688350 9623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:29.688432 9615 server_base.cc:1034] running on GCE node
I20260620 08:55:29.689116 9615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:29.690225 9615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:29.691397 9615 hybrid_clock.cc:648] HybridClock initialized: now 1781945729691384 us; error 29 us; skew 500 ppm
I20260620 08:55:29.693220 9615 webserver.cc:460] Webserver started at http://127.0.98.62:44135/ using document root <none> and password file <none>
I20260620 08:55:29.693774 9615 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:29.693843 9615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:29.694026 9615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:29.696033 9615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "55acfbc9f8fc486b90a8a8331bbaa50d"
format_stamp: "Formatted at 2026-06-20 08:55:29 on dist-test-slave-fb1v"
server_key: "f3261aa108690f377fc2ec85cd867a26"
server_key_iv: "9a87a11e55357588d3f6e5c717876cd5"
server_key_version: "encryptionkey@0"
I20260620 08:55:29.696537 9615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "55acfbc9f8fc486b90a8a8331bbaa50d"
format_stamp: "Formatted at 2026-06-20 08:55:29 on dist-test-slave-fb1v"
server_key: "f3261aa108690f377fc2ec85cd867a26"
server_key_iv: "9a87a11e55357588d3f6e5c717876cd5"
server_key_version: "encryptionkey@0"
I20260620 08:55:29.699949 9615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:55:29.702234 9629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:29.703431 9615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:29.703542 9615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "55acfbc9f8fc486b90a8a8331bbaa50d"
format_stamp: "Formatted at 2026-06-20 08:55:29 on dist-test-slave-fb1v"
server_key: "f3261aa108690f377fc2ec85cd867a26"
server_key_iv: "9a87a11e55357588d3f6e5c717876cd5"
server_key_version: "encryptionkey@0"
I20260620 08:55:29.703661 9615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:29.717926 9615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:29.718617 9615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:29.718798 9615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:29.725848 9615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38865
I20260620 08:55:29.725893 9681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38865 every 8 connection(s)
I20260620 08:55:29.726768 9615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:29.730000 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:29.735692 9682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d: Bootstrap starting.
I20260620 08:55:29.736353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9615
I20260620 08:55:29.736461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:29.736712 392 external_mini_cluster.cc:1428] Setting key d90c308b2243251d55e8c6afe7ac500c
I20260620 08:55:29.738304 9682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:29.738976 9682 log.cc:824] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:29.740876 9682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d: No bootstrap required, opened a new log
I20260620 08:55:29.745239 9682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 08:55:29.745522 9682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:29.745609 9682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55acfbc9f8fc486b90a8a8331bbaa50d, State: Initialized, Role: FOLLOWER
I20260620 08:55:29.746079 9682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 08:55:29.746235 9682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:29.746378 9682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:29.746470 9682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:29.746861 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:29.737789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.0.98.62:38865)
0620 08:55:29.738503 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:29.738525 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:29.738568 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:55:29.739205 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:29.739230 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:29.739672 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:29.740000 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:29.740400 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.741774 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.743062 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.743800 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.743990 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.744770 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:29.744784 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:29.744801 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:29.745263 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:29.745331 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:29.745356 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:29.745452 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:29.745617 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 08:55:29.745905 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260620 08:55:29.747469 9682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 08:55:29.747788 9682 leader_election.cc:302] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55acfbc9f8fc486b90a8a8331bbaa50d; no voters:
I20260620 08:55:29.748066 9682 leader_election.cc:288] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:29.748171 9687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:29.748387 9687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [term 1 LEADER]: Becoming Leader. State: Replica: 55acfbc9f8fc486b90a8a8331bbaa50d, State: Running, Role: LEADER
I20260620 08:55:29.748744 9687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 08:55:29.749488 9682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:29.750007 9687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55acfbc9f8fc486b90a8a8331bbaa50d. Latest consensus state: current_term: 1 leader_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } } }
I20260620 08:55:29.750157 9687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:29.750515 9688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55acfbc9f8fc486b90a8a8331bbaa50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } } }
I20260620 08:55:29.750644 9688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:29.750569 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:29.754038 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:29.759765 9693 catalog_manager.cc:1347] Generated new cluster ID: d18dce5c92114bbc98a5d79fb197c118
I20260620 08:55:29.759835 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:29.779170 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:29.780215 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:29.785305 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55acfbc9f8fc486b90a8a8331bbaa50d: Generated new TSK 0
I20260620 08:55:29.785966 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:55:29.797119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 08:55:29.904410 9706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:29.904649 9706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:29.904682 9706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:29.908691 9706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:29.908756 9706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:29.908813 9706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:29.913762 9706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:29.914793 9706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:29.916128 9706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:29.925385 9714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:29.925422 9711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:29.925385 9712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:29.925631 9706 server_base.cc:1034] running on GCE node
I20260620 08:55:29.926057 9706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:29.926630 9706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:29.927855 9706 hybrid_clock.cc:648] HybridClock initialized: now 1781945729927833 us; error 38 us; skew 500 ppm
I20260620 08:55:29.929602 9706 webserver.cc:460] Webserver started at http://127.0.98.1:42069/ using document root <none> and password file <none>
I20260620 08:55:29.930114 9706 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:29.930179 9706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:29.930331 9706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:29.932327 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "9cbcebfa53ef452b8a7b99c45cefdbb1"
format_stamp: "Formatted at 2026-06-20 08:55:29 on dist-test-slave-fb1v"
server_key: "8fe6acbf2abd24a110a97b5c6321d0a6"
server_key_iv: "512ef8fb3b5bc2d94b1f630f9a84eb6c"
server_key_version: "encryptionkey@0"
I20260620 08:55:29.932806 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9cbcebfa53ef452b8a7b99c45cefdbb1"
format_stamp: "Formatted at 2026-06-20 08:55:29 on dist-test-slave-fb1v"
server_key: "8fe6acbf2abd24a110a97b5c6321d0a6"
server_key_iv: "512ef8fb3b5bc2d94b1f630f9a84eb6c"
server_key_version: "encryptionkey@0"
I20260620 08:55:29.936319 9706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:29.938827 9720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:29.940060 9706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:29.940165 9706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "9cbcebfa53ef452b8a7b99c45cefdbb1"
format_stamp: "Formatted at 2026-06-20 08:55:29 on dist-test-slave-fb1v"
server_key: "8fe6acbf2abd24a110a97b5c6321d0a6"
server_key_iv: "512ef8fb3b5bc2d94b1f630f9a84eb6c"
server_key_version: "encryptionkey@0"
I20260620 08:55:29.940253 9706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:29.959506 9706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:29.960193 9706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:29.960342 9706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:29.960865 9706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:29.961742 9706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:29.961797 9706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:29.961836 9706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:29.961850 9706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:29.970348 9706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36155
I20260620 08:55:29.970655 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36155 every 8 connection(s)
I20260620 08:55:29.971285 9706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:29.973170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9706
I20260620 08:55:29.973289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:29.973551 392 external_mini_cluster.cc:1428] Setting key a5cc869500970e8b3a835176490bfa8c
I20260620 08:55:29.975251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 08:55:29.980700 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:29.973070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57685 (local address 127.0.98.62:38865)
0620 08:55:29.973214 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:29.973219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:29.974314 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 08:55:29.975727 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:29.975730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:29.975783 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:29.975924 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:29.976844 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.977427 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.978609 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.978961 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.979082 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.980228 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:29.980232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:29.980235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:29.980250 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:29.980273 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:29.980284 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:29.980366 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:29.980484 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 08:55:29.980547 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:55:29.981490 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:29.973478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.1:57685)
0620 08:55:29.974168 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:29.974219 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:29.975489 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:29.976116 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:29.976123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:29.976211 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:29.976708 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:29.976720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.977563 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.977567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:29.978461 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:29.978473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:29.979134 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:29.979137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:29.979292 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:29.979930 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:55:29.979932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:29.979987 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:55:29.980069 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:29.980363 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:29.980368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:29.980371 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:55:29.980461 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:55:29.980684 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":117,"threads_started":1}
I20260620 08:55:29.982887 9834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260620 08:55:29.983095 9834 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:29.983594 9834 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260620 08:55:29.985251 9646 ts_manager.cc:194] Registered new tserver with Master: 9cbcebfa53ef452b8a7b99c45cefdbb1 (127.0.98.1:36155)
I20260620 08:55:29.986752 9646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57685
W20260620 08:55:30.090449 9837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:30.090754 9837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:30.090802 9837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:30.094978 9837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:30.095036 9837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:30.095090 9837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:30.101269 9837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:30.102727 9837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:30.104362 9837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:30.114507 9842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.114516 9845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.114617 9837 server_base.cc:1034] running on GCE node
W20260620 08:55:30.114516 9843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.115442 9837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:30.116158 9837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:30.117408 9837 hybrid_clock.cc:648] HybridClock initialized: now 1781945730117379 us; error 67 us; skew 500 ppm
I20260620 08:55:30.119251 9837 webserver.cc:460] Webserver started at http://127.0.98.2:44917/ using document root <none> and password file <none>
I20260620 08:55:30.119843 9837 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:30.119910 9837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:30.120103 9837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:30.122253 9837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "8dcc4479e11b4e3d9b40efa884ceeb1d"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "e200ba7e123ddf68be5692c1c6230ae6"
server_key_iv: "6505cf347c553c274edd220d029929b7"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.122772 9837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8dcc4479e11b4e3d9b40efa884ceeb1d"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "e200ba7e123ddf68be5692c1c6230ae6"
server_key_iv: "6505cf347c553c274edd220d029929b7"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.126796 9837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 08:55:30.129415 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.130681 9837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:30.130820 9837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "8dcc4479e11b4e3d9b40efa884ceeb1d"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "e200ba7e123ddf68be5692c1c6230ae6"
server_key_iv: "6505cf347c553c274edd220d029929b7"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.130923 9837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:30.145671 9837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:30.146337 9837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:30.146497 9837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:30.147049 9837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:30.147974 9837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:30.148023 9837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.148073 9837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:30.148102 9837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.156446 9837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37483
I20260620 08:55:30.156479 9964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37483 every 8 connection(s)
I20260620 08:55:30.157311 9837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:30.161768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9837
I20260620 08:55:30.161887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:55:30.162132 392 external_mini_cluster.cc:1428] Setting key c82a90543817f542947cb8ebec0920cc
I20260620 08:55:30.163834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 08:55:30.166775 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.159049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41211 (local address 127.0.98.62:38865)
0620 08:55:30.159192 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:30.159195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:30.160192 (+ 997us) server_negotiation.cc:407] Connection header received
0620 08:55:30.161520 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:30.161523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:30.161579 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:30.161650 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:30.162578 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.163122 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.164371 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.164580 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.164665 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:30.166325 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:30.166329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:30.166332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:30.166345 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:30.166364 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:30.166371 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:30.166431 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:30.166568 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 08:55:30.166628 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 08:55:30.167799 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.159431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.2:41211)
0620 08:55:30.160052 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:30.160089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:30.161328 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:30.161823 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:30.161830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:30.161919 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:30.162421 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:30.162438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.163267 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.163270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:30.164257 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:30.164267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.164715 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.164719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:30.165070 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:30.166052 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:55:30.166054 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:30.166095 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:55:30.166161 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:30.166443 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:30.166452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:30.166458 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:55:30.166561 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:55:30.166855 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":134,"threads_started":1}
I20260620 08:55:30.168974 9965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260620 08:55:30.169194 9965 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:30.169715 9965 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260620 08:55:30.170713 9646 ts_manager.cc:194] Registered new tserver with Master: 8dcc4479e11b4e3d9b40efa884ceeb1d (127.0.98.2:37483)
I20260620 08:55:30.171442 9646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41211
W20260620 08:55:30.271363 9968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:30.271627 9968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:30.271679 9968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:30.275573 9968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:30.275645 9968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:30.275717 9968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:30.280325 9968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:30.281302 9968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:30.282552 9968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:30.291652 9974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.291651 9973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.291669 9976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.291908 9968 server_base.cc:1034] running on GCE node
I20260620 08:55:30.292373 9968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:30.292968 9968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:30.294152 9968 hybrid_clock.cc:648] HybridClock initialized: now 1781945730294125 us; error 42 us; skew 500 ppm
I20260620 08:55:30.295957 9968 webserver.cc:460] Webserver started at http://127.0.98.3:38941/ using document root <none> and password file <none>
I20260620 08:55:30.296481 9968 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:30.296545 9968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:30.296744 9968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:30.298784 9968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "5da08348cf514530bb5b86ceed032e6c"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "71a0b14cb37f12ad6a18f003315d0dd0"
server_key_iv: "96e7e4add1d6a3c5121217565a02abe0"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.299281 9968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5da08348cf514530bb5b86ceed032e6c"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "71a0b14cb37f12ad6a18f003315d0dd0"
server_key_iv: "96e7e4add1d6a3c5121217565a02abe0"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.302706 9968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:55:30.305065 9982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.306185 9968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:55:30.306295 9968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "5da08348cf514530bb5b86ceed032e6c"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "71a0b14cb37f12ad6a18f003315d0dd0"
server_key_iv: "96e7e4add1d6a3c5121217565a02abe0"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.306404 9968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:30.332095 9968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:30.332783 9968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:30.332952 9968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:30.333536 9968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:30.334434 9968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:30.334483 9968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.334602 9968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:30.334637 9968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.344290 9968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36125
I20260620 08:55:30.344354 10095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36125 every 8 connection(s)
I20260620 08:55:30.345273 9968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:30.349468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 9968
I20260620 08:55:30.349550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:30.349828 392 external_mini_cluster.cc:1428] Setting key 5b8a9b66995538874032da291b7727fa
I20260620 08:55:30.354693 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.347090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48993 (local address 127.0.98.62:38865)
0620 08:55:30.347216 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:30.347220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:30.348392 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 08:55:30.349826 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:30.349882 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:30.349953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:30.350046 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:30.351044 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.351709 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.352888 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.353135 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.353233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:30.354266 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:30.354269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:30.354272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:30.354284 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:30.354303 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:30.354317 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:30.354374 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:30.354504 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 08:55:30.354560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:55:30.356118 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.347527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.3:48993)
0620 08:55:30.348197 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:30.348235 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:30.349606 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:30.350230 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:30.350239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:30.350338 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:30.350906 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:30.350924 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.351854 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.351857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:30.352727 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:30.352737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:30.353281 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:30.353285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:30.353456 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:30.354034 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:55:30.354036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:30.354072 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:55:30.354132 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:30.354385 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:30.354393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:30.354399 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:55:30.354499 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:55:30.354761 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":122,"threads_started":1}
I20260620 08:55:30.357123 10096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260620 08:55:30.357336 10096 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:30.357781 10096 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260620 08:55:30.358680 9646 ts_manager.cc:194] Registered new tserver with Master: 5da08348cf514530bb5b86ceed032e6c (127.0.98.3:36125)
I20260620 08:55:30.359263 9646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48993
I20260620 08:55:30.363118 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:55:30.365720 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:30.364918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.0.98.62:38865)
0620 08:55:30.365034 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:30.365038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:30.365207 (+ 169us) server_negotiation.cc:407] Connection header received
0620 08:55:30.365329 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:30.365331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:30.365364 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:30.365425 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:30.365605 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36550: BlockingRecv error: recv got EOF from 127.0.0.1:36550 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:55:30.366155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9706
I20260620 08:55:30.370632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9837
I20260620 08:55:30.374897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9968
I20260620 08:55:30.379036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 9615
2026-06-20T08:55:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (844 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T08:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:30Z Disabled control of system clock
I20260620 08:55:30.405078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35169
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:55:30.511643 10107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:30.511893 10107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:30.511929 10107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:30.515789 10107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:30.515837 10107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:30.515852 10107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:30.515867 10107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:30.515880 10107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:30.520537 10107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:30.521498 10107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:30.522755 10107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:30.530637 10113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.530665 10112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.530690 10107 server_base.cc:1034] running on GCE node
W20260620 08:55:30.530710 10115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.531478 10107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:30.532521 10107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:30.533679 10107 hybrid_clock.cc:648] HybridClock initialized: now 1781945730533662 us; error 33 us; skew 500 ppm
I20260620 08:55:30.535442 10107 webserver.cc:460] Webserver started at http://127.0.98.62:42433/ using document root <none> and password file <none>
I20260620 08:55:30.535998 10107 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:30.536058 10107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:30.536229 10107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:30.538089 10107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "a2d65b85505e49ae839f207bdd652c24"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "3b0f349cf32fec94e3fa15ef06829384"
server_key_iv: "4139903110f3914e3963f9056f9ec0db"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.538580 10107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2d65b85505e49ae839f207bdd652c24"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "3b0f349cf32fec94e3fa15ef06829384"
server_key_iv: "4139903110f3914e3963f9056f9ec0db"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.541939 10107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:55:30.544312 10121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.545418 10107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:30.545523 10107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "a2d65b85505e49ae839f207bdd652c24"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "3b0f349cf32fec94e3fa15ef06829384"
server_key_iv: "4139903110f3914e3963f9056f9ec0db"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.545609 10107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:30.549832 10107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:30.550354 10107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:30.550495 10107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:30.556572 10107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35169
I20260620 08:55:30.556588 10173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35169 every 8 connection(s)
I20260620 08:55:30.557361 10107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:30.559865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10107
I20260620 08:55:30.559976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:30.560269 392 external_mini_cluster.cc:1428] Setting key 11251eb6d905c6bec9d03fc52ca8b9ae
I20260620 08:55:30.560343 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:30.565367 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.561490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.0.98.62:35169)
0620 08:55:30.562043 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:30.562057 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:30.562088 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:55:30.562765 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:30.562781 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:30.563113 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:30.563372 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:30.564166 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:30.564171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:30.564184 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:30.564195 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:30.564250 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:30.564266 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:30.564376 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:30.564471 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 08:55:30.564708 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260620 08:55:30.566638 10174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24: Bootstrap starting.
I20260620 08:55:30.568707 10174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:30.569332 10174 log.cc:824] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:30.571089 10174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24: No bootstrap required, opened a new log
I20260620 08:55:30.574239 10174 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d65b85505e49ae839f207bdd652c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260620 08:55:30.574429 10174 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:30.574461 10174 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d65b85505e49ae839f207bdd652c24, State: Initialized, Role: FOLLOWER
I20260620 08:55:30.574851 10174 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d65b85505e49ae839f207bdd652c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260620 08:55:30.574990 10174 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:30.575047 10174 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:30.575146 10174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:30.575961 10174 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d65b85505e49ae839f207bdd652c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260620 08:55:30.576256 10174 leader_election.cc:302] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d65b85505e49ae839f207bdd652c24; no voters:
I20260620 08:55:30.576508 10174 leader_election.cc:288] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:30.576635 10179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:30.576848 10179 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [term 1 LEADER]: Becoming Leader. State: Replica: a2d65b85505e49ae839f207bdd652c24, State: Running, Role: LEADER
I20260620 08:55:30.577109 10179 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d65b85505e49ae839f207bdd652c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260620 08:55:30.577551 10174 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:30.578493 10180 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d65b85505e49ae839f207bdd652c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d65b85505e49ae839f207bdd652c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } } }
I20260620 08:55:30.578611 10180 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:30.578946 10188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:30.579502 10181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d65b85505e49ae839f207bdd652c24. Latest consensus state: current_term: 1 leader_uuid: "a2d65b85505e49ae839f207bdd652c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d65b85505e49ae839f207bdd652c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } } }
I20260620 08:55:30.579622 10181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24 [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:30.582125 10188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:30.587257 10188 catalog_manager.cc:1347] Generated new cluster ID: c7b20801313a49bc85057c557707fb15
I20260620 08:55:30.587328 10188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:30.609949 10188 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:30.610255 10188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:30.624228 10188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d65b85505e49ae839f207bdd652c24: Generated new TSK 0
I20260620 08:55:30.624897 10188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:55:30.638912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:55:30.764978 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:30.765220 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:30.765267 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:30.769200 10198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:30.769253 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:30.769311 10198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:30.773940 10198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:30.774925 10198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:30.776204 10198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:30.785399 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.785415 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.785487 10198 server_base.cc:1034] running on GCE node
W20260620 08:55:30.785434 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.785995 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:30.786574 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:30.787706 10198 hybrid_clock.cc:648] HybridClock initialized: now 1781945730787684 us; error 44 us; skew 500 ppm
I20260620 08:55:30.789532 10198 webserver.cc:460] Webserver started at http://127.0.98.1:33743/ using document root <none> and password file <none>
I20260620 08:55:30.790062 10198 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:30.790124 10198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:30.790302 10198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:30.792304 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "637b094a06c8448992ddba0b1aba8857"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "61ab13412b96bebd83d62015bc6d3ecd"
server_key_iv: "3c1f62b8f40f70a5ee6ad58d24ba40d1"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.792807 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "637b094a06c8448992ddba0b1aba8857"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "61ab13412b96bebd83d62015bc6d3ecd"
server_key_iv: "3c1f62b8f40f70a5ee6ad58d24ba40d1"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.796192 10198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:55:30.798408 10212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.799518 10198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:30.799642 10198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "637b094a06c8448992ddba0b1aba8857"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "61ab13412b96bebd83d62015bc6d3ecd"
server_key_iv: "3c1f62b8f40f70a5ee6ad58d24ba40d1"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.799731 10198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:30.803958 10198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:30.804481 10198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:30.804623 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:30.805111 10198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:30.805910 10198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:30.805953 10198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.806000 10198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:30.806028 10198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.813705 10198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35567
I20260620 08:55:30.813727 10325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35567 every 8 connection(s)
I20260620 08:55:30.814492 10198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:55:30.815089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10198
I20260620 08:55:30.815197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:30.815436 392 external_mini_cluster.cc:1428] Setting key 4b81396b01bc9497a9fc0a3f964714e7
I20260620 08:55:30.817410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 08:55:30.821894 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.816131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59181 (local address 127.0.98.62:35169)
0620 08:55:30.817450 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:30.817453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:30.819012 (+ 1559us) server_negotiation.cc:407] Connection header received
0620 08:55:30.820273 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:30.820276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:30.820324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:30.820395 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:30.821468 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:30.821471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:30.821474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:30.821486 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:30.821507 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:30.821515 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:30.821588 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:30.821724 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 08:55:30.821765 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240}
I20260620 08:55:30.822741 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.816633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.1:59181)
0620 08:55:30.818881 (+ 2248us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:30.818919 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:30.820081 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:30.820548 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:30.820554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:30.820649 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:30.821220 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:55:30.821221 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:30.821264 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:55:30.821331 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:30.821580 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:30.821605 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:30.821611 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:55:30.821759 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 08:55:30.822057 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1860,"thread_start_us":130,"threads_started":1}
I20260620 08:55:30.824110 10326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35169
I20260620 08:55:30.824525 10326 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:30.825163 10326 heartbeater.cc:511] Master 127.0.98.62:35169 requested a full tablet report, sending...
I20260620 08:55:30.826864 10138 ts_manager.cc:194] Registered new tserver with Master: 637b094a06c8448992ddba0b1aba8857 (127.0.98.1:35567)
W20260620 08:55:30.923558 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:30.923857 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:30.923897 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:30.927806 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:30.927851 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:30.927904 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:30.932544 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:30.933508 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:30.934751 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:30.943682 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.943686 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:30.943712 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:30.943948 10329 server_base.cc:1034] running on GCE node
I20260620 08:55:30.944463 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:30.945062 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:30.946233 10329 hybrid_clock.cc:648] HybridClock initialized: now 1781945730946216 us; error 31 us; skew 500 ppm
I20260620 08:55:30.948004 10329 webserver.cc:460] Webserver started at http://127.0.98.2:37915/ using document root <none> and password file <none>
I20260620 08:55:30.948591 10329 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:30.948660 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:30.948851 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:30.950806 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "345172ed26554691b3576cdd378d7f6c"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "77112a18fe98feb53695615e95c407b0"
server_key_iv: "096b007d4a0bebf06c8c7b47b2517c5a"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.951329 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "345172ed26554691b3576cdd378d7f6c"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "77112a18fe98feb53695615e95c407b0"
server_key_iv: "096b007d4a0bebf06c8c7b47b2517c5a"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.954730 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:55:30.957353 10343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.958730 10329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 08:55:30.958842 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "345172ed26554691b3576cdd378d7f6c"
format_stamp: "Formatted at 2026-06-20 08:55:30 on dist-test-slave-fb1v"
server_key: "77112a18fe98feb53695615e95c407b0"
server_key_iv: "096b007d4a0bebf06c8c7b47b2517c5a"
server_key_version: "encryptionkey@0"
I20260620 08:55:30.958940 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:30.963312 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:30.963903 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:30.964052 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:30.964551 10329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:30.965406 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:30.965451 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.965499 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:30.965528 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:30.973626 10329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43615
I20260620 08:55:30.973675 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43615 every 8 connection(s)
I20260620 08:55:30.974617 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:30.980933 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.976667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54183 (local address 127.0.98.62:35169)
0620 08:55:30.976805 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:30.976809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:30.977952 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 08:55:30.979301 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:30.979304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:30.979357 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:30.979435 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:30.980511 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:30.980515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:30.980518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:30.980529 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:30.980551 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:30.980559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:30.980635 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:30.980751 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 08:55:30.980795 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:55:30.981899 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:30.977059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.2:54183)
0620 08:55:30.977794 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:30.977844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:30.979105 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:30.979587 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:30.979593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:30.979726 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:30.980263 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:55:30.980265 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:30.980303 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:55:30.980371 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:30.980626 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:30.980633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:30.980638 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:55:30.980788 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 08:55:30.981093 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":142,"threads_started":1}
I20260620 08:55:30.982882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10329
I20260620 08:55:30.983002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:55:30.983412 392 external_mini_cluster.cc:1428] Setting key 5d3b0032d4b2d49f1cbf4b74bfee2d9a
I20260620 08:55:30.983743 10457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35169
I20260620 08:55:30.983983 10457 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:30.984382 10457 heartbeater.cc:511] Master 127.0.98.62:35169 requested a full tablet report, sending...
I20260620 08:55:30.985271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 08:55:30.985388 10138 ts_manager.cc:194] Registered new tserver with Master: 345172ed26554691b3576cdd378d7f6c (127.0.98.2:43615)
W20260620 08:55:31.095999 10460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:31.096259 10460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:31.096298 10460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:31.100291 10460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:31.100342 10460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:31.100396 10460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:31.105381 10460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:31.106384 10460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:31.107682 10460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:31.117336 10465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:31.117347 10466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:31.117390 10460 server_base.cc:1034] running on GCE node
W20260620 08:55:31.117341 10468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:31.118006 10460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:31.118600 10460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:31.119774 10460 hybrid_clock.cc:648] HybridClock initialized: now 1781945731119740 us; error 31 us; skew 500 ppm
I20260620 08:55:31.121887 10460 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260620 08:55:31.122438 10460 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:31.122503 10460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:31.122676 10460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:31.124696 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f25518c6ef44753a32b2e6ed3fd8b58"
format_stamp: "Formatted at 2026-06-20 08:55:31 on dist-test-slave-fb1v"
server_key: "d5a6a2be77e04078052d41e372498638"
server_key_iv: "010f9f556dc72d3bcd9924c9e695a1bf"
server_key_version: "encryptionkey@0"
I20260620 08:55:31.125201 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f25518c6ef44753a32b2e6ed3fd8b58"
format_stamp: "Formatted at 2026-06-20 08:55:31 on dist-test-slave-fb1v"
server_key: "d5a6a2be77e04078052d41e372498638"
server_key_iv: "010f9f556dc72d3bcd9924c9e695a1bf"
server_key_version: "encryptionkey@0"
I20260620 08:55:31.129031 10460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:55:31.131323 10474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:31.132547 10460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:55:31.132651 10460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "8f25518c6ef44753a32b2e6ed3fd8b58"
format_stamp: "Formatted at 2026-06-20 08:55:31 on dist-test-slave-fb1v"
server_key: "d5a6a2be77e04078052d41e372498638"
server_key_iv: "010f9f556dc72d3bcd9924c9e695a1bf"
server_key_version: "encryptionkey@0"
I20260620 08:55:31.132745 10460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:31.137058 10460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:31.137598 10460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:31.137738 10460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:31.138217 10460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:31.139045 10460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:31.139103 10460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:31.139150 10460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:31.139174 10460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:31.147541 10460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36187
I20260620 08:55:31.147571 10587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36187 every 8 connection(s)
I20260620 08:55:31.148447 10460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:55:31.151046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10460
I20260620 08:55:31.151146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:31.151482 392 external_mini_cluster.cc:1428] Setting key ff8c88945dca6a522f076bc95863ac12
I20260620 08:55:31.154433 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:31.150182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51267 (local address 127.0.98.62:35169)
0620 08:55:31.150309 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:31.150313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:31.151429 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 08:55:31.152962 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:31.152965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:31.153002 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:31.153084 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:31.154095 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:31.154098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:31.154100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:55:31.154110 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:31.154129 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:55:31.154137 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:31.154213 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:31.154306 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 08:55:31.154338 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:55:31.155205 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:31.150517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.3:51267)
0620 08:55:31.151258 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:31.151311 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:31.152768 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:31.153208 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:31.153215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:31.153323 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:31.153837 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:55:31.153839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:31.153875 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:55:31.153948 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:31.154210 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:31.154214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:31.154217 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:55:31.154332 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 08:55:31.154505 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260620 08:55:31.156138 10588 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35169
I20260620 08:55:31.156317 10588 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:31.156643 10588 heartbeater.cc:511] Master 127.0.98.62:35169 requested a full tablet report, sending...
I20260620 08:55:31.157500 10138 ts_manager.cc:194] Registered new tserver with Master: 8f25518c6ef44753a32b2e6ed3fd8b58 (127.0.98.3:36187)
I20260620 08:55:31.165073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:55:31.167640 10177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:55:31.166859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.98.62:35169)
0620 08:55:31.166994 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:31.166998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:31.167160 (+ 162us) server_negotiation.cc:407] Connection header received
0620 08:55:31.167223 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:31.167226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:31.167254 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:31.167335 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:31.167504 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48536: BlockingRecv error: recv got EOF from 127.0.0.1:48536 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 08:55:31.167778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10198
I20260620 08:55:31.172207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10329
I20260620 08:55:31.176460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10460
I20260620 08:55:31.180840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10107
2026-06-20T08:55:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (802 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:31 dist-test-slave-fb1v krb5kdc[10598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:31 dist-test-slave-fb1v krb5kdc[10598](info): set up 2 sockets
Jun 20 08:55:31 dist-test-slave-fb1v krb5kdc[10598](info): commencing operation
krb5kdc: starting...
W20260620 08:55:32.140406 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:55:32.258566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:32.363832 10611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:32.364081 10611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:32.364120 10611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:32.368099 10611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:55:32.368153 10611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:32.368170 10611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:32.368185 10611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:55:32.368199 10611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:55:32.373314 10611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:32.374293 10611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:32.375238 10611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:32.383117 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:32.383188 10617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:32.383157 10616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:32.383288 10611 server_base.cc:1034] running on GCE node
I20260620 08:55:32.383919 10611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:32.384965 10611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:32.386157 10611 hybrid_clock.cc:648] HybridClock initialized: now 1781945732386111 us; error 59 us; skew 500 ppm
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:32.388799 10611 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:55:32.389611 10611 webserver.cc:460] Webserver started at http://127.0.98.62:41665/ using document root <none> and password file <none>
I20260620 08:55:32.390128 10611 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:32.390185 10611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:32.390364 10611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:32.392287 10611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "221aa5b2ddae4814b4f4725c5c7973db"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "a1fa7b5d2ee800b54c1cf6e2cb0f70f3"
server_key_iv: "528f54aafa669a91ea598184511af46a"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.392779 10611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "221aa5b2ddae4814b4f4725c5c7973db"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "a1fa7b5d2ee800b54c1cf6e2cb0f70f3"
server_key_iv: "528f54aafa669a91ea598184511af46a"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.396378 10611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:55:32.398743 10626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.400017 10611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:32.400115 10611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "221aa5b2ddae4814b4f4725c5c7973db"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "a1fa7b5d2ee800b54c1cf6e2cb0f70f3"
server_key_iv: "528f54aafa669a91ea598184511af46a"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.400211 10611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:32.451735 10611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:32.454605 10611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:32.454771 10611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:32.461355 10611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37055
I20260620 08:55:32.461601 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37055 every 8 connection(s)
I20260620 08:55:32.462284 10611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:55:32.464958 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:32.465126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10611
I20260620 08:55:32.465245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:55:32.465499 392 external_mini_cluster.cc:1428] Setting key 8bd0517704c22a9f6636dcc8e1255ad9
I20260620 08:55:32.472069 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: Bootstrap starting.
I20260620 08:55:32.474449 10679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: Neither blocks nor log segments found. Creating new log.
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:32.475169 10679 log.cc:824] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:32.477414 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: No bootstrap required, opened a new log
I20260620 08:55:32.479884 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:32.466831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.0.98.62:37055)
0620 08:55:32.467240 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:32.467258 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:32.467299 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:55:32.468187 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:32.468211 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:32.468614 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:32.469053 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:32.469682 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.471150 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.472062 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.472701 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.472857 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:32.475186 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:32.475193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:32.475209 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:32.475240 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:32.477180 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:32.477628 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:32.477634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:32.477640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:32.477753 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:32.478002 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:32.478006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:32.478007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:32.478399 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:32.478574 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:32.478853 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 08:55:32.479123 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260620 08:55:32.480901 10679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221aa5b2ddae4814b4f4725c5c7973db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 08:55:32.481106 10679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:32.481161 10679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221aa5b2ddae4814b4f4725c5c7973db, State: Initialized, Role: FOLLOWER
I20260620 08:55:32.481540 10679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221aa5b2ddae4814b4f4725c5c7973db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 08:55:32.481657 10679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:55:32.481716 10679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:55:32.481809 10679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:32.482683 10679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221aa5b2ddae4814b4f4725c5c7973db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 08:55:32.482980 10679 leader_election.cc:302] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221aa5b2ddae4814b4f4725c5c7973db; no voters:
I20260620 08:55:32.483285 10679 leader_election.cc:288] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:55:32.483450 10684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:32.483685 10684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [term 1 LEADER]: Becoming Leader. State: Replica: 221aa5b2ddae4814b4f4725c5c7973db, State: Running, Role: LEADER
I20260620 08:55:32.483981 10684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221aa5b2ddae4814b4f4725c5c7973db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 08:55:32.484462 10679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:55:32.485020 10684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221aa5b2ddae4814b4f4725c5c7973db. Latest consensus state: current_term: 1 leader_uuid: "221aa5b2ddae4814b4f4725c5c7973db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221aa5b2ddae4814b4f4725c5c7973db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260620 08:55:32.485133 10684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [sys.catalog]: This master's current role is: LEADER
I20260620 08:55:32.485548 10685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221aa5b2ddae4814b4f4725c5c7973db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221aa5b2ddae4814b4f4725c5c7973db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260620 08:55:32.485649 10685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db [sys.catalog]: This master's current role is: LEADER
W20260620 08:55:32.488886 10699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:55:32.488970 10699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:55:32.489027 10691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:55:32.489665 10691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:55:32.494184 10691 catalog_manager.cc:1347] Generated new cluster ID: be07f64e024b4ee2bc065917ff94a655
I20260620 08:55:32.494239 10691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:55:32.506798 10691 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:55:32.508041 10691 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:55:32.514334 10691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: Generated new TSK 0
I20260620 08:55:32.515108 10691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:55:32.546752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:32.651448 10705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:32.651739 10705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:32.651786 10705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:32.655690 10705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:32.655742 10705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:32.655800 10705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:55:32.660965 10705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:32.661938 10705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:32.662889 10705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:32.671896 10711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:32.671893 10710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:32.672036 10705 server_base.cc:1034] running on GCE node
W20260620 08:55:32.671893 10713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:32.672497 10705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:32.673067 10705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:32.674263 10705 hybrid_clock.cc:648] HybridClock initialized: now 1781945732674246 us; error 38 us; skew 500 ppm
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:32.676924 10705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:55:32.677745 10705 webserver.cc:460] Webserver started at http://127.0.98.1:34131/ using document root <none> and password file <none>
I20260620 08:55:32.678261 10705 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:32.678331 10705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:32.678519 10705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:32.680470 10705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "71f8f7abad1292fa6abe0541cf549b76"
server_key_iv: "90dcfa6e9b63c0e1c5ff2d22ae844f51"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.680970 10705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "71f8f7abad1292fa6abe0541cf549b76"
server_key_iv: "90dcfa6e9b63c0e1c5ff2d22ae844f51"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.684536 10705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:55:32.686892 10720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.688086 10705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:32.688182 10705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "71f8f7abad1292fa6abe0541cf549b76"
server_key_iv: "90dcfa6e9b63c0e1c5ff2d22ae844f51"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.688267 10705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:32.709884 10705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:32.712872 10705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:32.713042 10705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:32.713603 10705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:32.714395 10705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:32.714444 10705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.714495 10705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:32.714524 10705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.722990 10705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42431
I20260620 08:55:32.723081 10833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42431 every 8 connection(s)
I20260620 08:55:32.723862 10705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:32.733129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10705
I20260620 08:55:32.733233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:55:32.733479 392 external_mini_cluster.cc:1428] Setting key 5bd2dd818738b8d040942f6be57eb15c
I20260620 08:55:32.736538 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:32.725534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34433 (local address 127.0.98.62:37055)
0620 08:55:32.725651 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:32.725654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:32.726424 (+ 770us) server_negotiation.cc:407] Connection header received
0620 08:55:32.727671 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:32.727675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:32.727728 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:32.727818 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:32.728930 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.729441 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.730526 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.730744 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.730838 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:32.733308 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:32.733312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:32.733314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:32.733340 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:32.734634 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:32.735096 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:32.735100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:32.735101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:32.735174 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:32.735473 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:32.735476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:32.735478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:32.735689 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:32.735775 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:32.736253 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 08:55:32.736381 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:32.737332 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:32.725846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.1:34433)
0620 08:55:32.726276 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:32.726319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:32.727468 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:32.727985 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:32.727992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:32.728335 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:32.728789 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:32.728802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.729552 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.729555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:32.730405 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:32.730415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.730881 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.730884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:32.731038 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:32.731562 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:32.731578 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:32.733093 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:32.734740 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:32.734746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:32.734761 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:32.735004 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:32.735253 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:32.735256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:32.735258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:32.735388 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:32.735790 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:32.735797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:32.736032 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 08:55:32.736199 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 08:55:32.736509 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260620 08:55:32.738449 10834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260620 08:55:32.738692 10834 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:32.739249 10834 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260620 08:55:32.740906 10643 ts_manager.cc:194] Registered new tserver with Master: fccf245a21ff4b92a8ac2d8b4ae7be79 (127.0.98.1:42431)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:55:32.742415 10643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:55:32.758530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:32.864225 10839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:32.864492 10839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:32.864535 10839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:32.868522 10839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:32.868585 10839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:32.868667 10839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:55:32.873803 10839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:32.874789 10839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:32.875777 10839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:55:32.884864 10844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:32.885007 10839 server_base.cc:1034] running on GCE node
W20260620 08:55:32.884960 10845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:32.884864 10847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:32.885497 10839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:32.886168 10839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:32.887341 10839 hybrid_clock.cc:648] HybridClock initialized: now 1781945732887321 us; error 38 us; skew 500 ppm
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:32.889829 10839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:55:32.890597 10839 webserver.cc:460] Webserver started at http://127.0.98.2:46257/ using document root <none> and password file <none>
I20260620 08:55:32.891116 10839 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:32.891172 10839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:32.891331 10839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:32.893301 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7d7513239e2461e9e2b34ff14882d8a"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "82b1c3005c24a26cfbe591d68e3a3bb5"
server_key_iv: "5bf1ce352b9982d4aacbf6f201955d64"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.893774 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7d7513239e2461e9e2b34ff14882d8a"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "82b1c3005c24a26cfbe591d68e3a3bb5"
server_key_iv: "5bf1ce352b9982d4aacbf6f201955d64"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.897315 10839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:55:32.899776 10854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.900941 10839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:55:32.901038 10839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "d7d7513239e2461e9e2b34ff14882d8a"
format_stamp: "Formatted at 2026-06-20 08:55:32 on dist-test-slave-fb1v"
server_key: "82b1c3005c24a26cfbe591d68e3a3bb5"
server_key_iv: "5bf1ce352b9982d4aacbf6f201955d64"
server_key_version: "encryptionkey@0"
I20260620 08:55:32.901147 10839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:32.919590 10839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:32.922622 10839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:32.922797 10839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:32.923403 10839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:32.924247 10839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:32.924293 10839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.924346 10839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:32.924374 10839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:32.932565 10839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46391
I20260620 08:55:32.932613 10967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46391 every 8 connection(s)
I20260620 08:55:32.933421 10839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:55:32.934192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10839
I20260620 08:55:32.934296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:55:32.934587 392 external_mini_cluster.cc:1428] Setting key a89be92a760e8846d1cfbbfca410119f
Jun 20 08:55:32 dist-test-slave-fb1v krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:55:32.947187 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:32.935294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39389 (local address 127.0.98.62:37055)
0620 08:55:32.935415 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:32.935418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:32.936425 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 08:55:32.937804 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:32.937807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:32.937851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:32.937916 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:32.939331 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.939926 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.941128 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.941356 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.941445 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:32.943771 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:32.943775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:32.943778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:32.943803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:32.945273 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:32.945776 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:32.945780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:32.945781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:32.945858 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:32.946157 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:32.946161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:32.946162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:32.946349 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:32.946432 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:32.946911 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 08:55:32.947024 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:55:32.948011 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:32.935799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.2:39389)
0620 08:55:32.936274 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:32.936324 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:32.937590 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:32.938087 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:32.938096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:32.938589 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:32.939186 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:32.939203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.940046 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.940050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:32.941010 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:32.941019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:32.941490 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:32.941494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:32.941660 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:32.942201 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:32.942218 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:32.943591 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:32.945371 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:32.945377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:32.945390 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:32.945689 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:32.945941 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:32.945944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:32.945946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:32.946075 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:32.946446 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:32.946453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:32.946681 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 08:55:32.946858 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 08:55:32.947164 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260620 08:55:32.949220 10968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260620 08:55:32.949452 10968 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:32.949996 10968 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260620 08:55:32.951108 10643 ts_manager.cc:194] Registered new tserver with Master: d7d7513239e2461e9e2b34ff14882d8a (127.0.98.2:46391)
I20260620 08:55:32.951716 10643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:55:32.962600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:55:33.071885 10973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:55:33.072124 10973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:55:33.072165 10973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:55:33.076221 10973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:55:33.076277 10973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:55:33.076350 10973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:55:33.081691 10973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:55:33.082670 10973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:55:33.083673 10973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:55:33.094362 10973 server_base.cc:1034] running on GCE node
W20260620 08:55:33.094389 10979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:33.094586 10981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:55:33.094635 10978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:55:33.094990 10973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:55:33.095592 10973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:55:33.096799 10973 hybrid_clock.cc:648] HybridClock initialized: now 1781945733096780 us; error 33 us; skew 500 ppm
Jun 20 08:55:33 dist-test-slave-fb1v krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:55:33.099426 10973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:55:33.100418 10973 webserver.cc:460] Webserver started at http://127.0.98.3:43075/ using document root <none> and password file <none>
I20260620 08:55:33.100986 10973 fs_manager.cc:359] Metadata directory not provided
I20260620 08:55:33.101087 10973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:55:33.101333 10973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:55:33.103854 10973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "629f4911de294559af1e701ea6b72159"
format_stamp: "Formatted at 2026-06-20 08:55:33 on dist-test-slave-fb1v"
server_key: "da1741b44507c57d31d85e87f8b7dd71"
server_key_iv: "94703111806a4d01b277292cd86cc55a"
server_key_version: "encryptionkey@0"
I20260620 08:55:33.104403 10973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "629f4911de294559af1e701ea6b72159"
format_stamp: "Formatted at 2026-06-20 08:55:33 on dist-test-slave-fb1v"
server_key: "da1741b44507c57d31d85e87f8b7dd71"
server_key_iv: "94703111806a4d01b277292cd86cc55a"
server_key_version: "encryptionkey@0"
I20260620 08:55:33.108187 10973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:55:33.110522 10988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:33.111737 10973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:55:33.111853 10973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "629f4911de294559af1e701ea6b72159"
format_stamp: "Formatted at 2026-06-20 08:55:33 on dist-test-slave-fb1v"
server_key: "da1741b44507c57d31d85e87f8b7dd71"
server_key_iv: "94703111806a4d01b277292cd86cc55a"
server_key_version: "encryptionkey@0"
I20260620 08:55:33.111949 10973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:55:33.144798 10973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:55:33.147923 10973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:55:33.148103 10973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:55:33.148690 10973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:55:33.149546 10973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:55:33.149595 10973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:33.149647 10973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:55:33.149675 10973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:55:33.158094 10973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38839
I20260620 08:55:33.158330 11101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38839 every 8 connection(s)
I20260620 08:55:33.158987 10973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:55:33 dist-test-slave-fb1v krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:55:33.168985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 10973
I20260620 08:55:33.169088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:55:33.169363 392 external_mini_cluster.cc:1428] Setting key f03d6b9e6f2def571bf274add29df75b
I20260620 08:55:33.172865 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.160909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42061 (local address 127.0.98.62:37055)
0620 08:55:33.161060 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.161063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.161887 (+ 824us) server_negotiation.cc:407] Connection header received
0620 08:55:33.163133 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.163137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.163193 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.163263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:33.164471 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.165086 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.166250 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.166467 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.166552 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.169152 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:33.169156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:33.169159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:33.169187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:33.170960 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:33.171445 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:33.171448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:33.171450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:33.171530 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:33.171858 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:33.171861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:33.171863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:33.172041 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:33.172123 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.172601 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.172711 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 08:55:33.173630 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.161273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.3:42061)
0620 08:55:33.161715 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.161757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:33.162915 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:33.163460 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:33.163467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:33.163861 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:55:33.164319 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.164331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.165224 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.165227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.166127 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.166137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.166599 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.166603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.166754 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.167511 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:55:33.167531 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:55:33.168985 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:55:33.171064 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:33.171069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:33.171084 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:33.171370 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:33.171628 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:55:33.171631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:55:33.171633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:55:33.171763 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:55:33.172133 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:55:33.172140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:55:33.172377 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 08:55:33.172530 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 08:55:33.172853 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260620 08:55:33.174837 11102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260620 08:55:33.175047 11102 heartbeater.cc:464] Registering TS with master...
I20260620 08:55:33.175493 11102 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260620 08:55:33.176492 10643 ts_manager.cc:194] Registered new tserver with Master: 629f4911de294559af1e701ea6b72159 (127.0.98.3:38839)
I20260620 08:55:33.177026 10643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42061
I20260620 08:55:33.183391 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:55:33.193950 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.185332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.0.98.62:37055)
0620 08:55:33.185490 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.185494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.185622 (+ 128us) server_negotiation.cc:407] Connection header received
0620 08:55:33.185755 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.185758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.185800 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.185867 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:55:33.186558 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.187150 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.188102 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.188307 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.188417 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.189987 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:55:33.189992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:55:33.189994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:55:33.190041 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:55:33.191965 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:33.192426 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:33.192432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:33.192434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:33.192616 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:55:33.193045 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:55:33.193049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:55:33.193050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:55:33.193304 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:55:33.193417 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.193675 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.193790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:55:33.197798 10643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:55:33.200743 10643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:55:33.214933 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.210006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.0.98.3:38839)
0620 08:55:33.210617 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.210641 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.210663 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:55:33.210801 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.210807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.210984 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.211134 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:33.211269 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.212391 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.213685 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.214218 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.214310 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.214488 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.214694 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.214788 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":164,"threads_started":1}
I20260620 08:55:33.214924 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.209511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38839 (local address 127.0.0.1:46052)
0620 08:55:33.210591 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.210603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:33.210686 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:33.211117 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:33.211119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:33.211132 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:33.211198 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.211203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.212523 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.212526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.213573 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.213583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.214323 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.214327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.214491 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.214558 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 08:55:33.214660 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:55:33.214784 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":104,"thread_start_us":85,"threads_started":1}
I20260620 08:55:33.215008 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.209969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.0.98.1:42431)
0620 08:55:33.210270 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.210285 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.210373 (+ 88us) server_negotiation.cc:407] Connection header received
0620 08:55:33.210480 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.210484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.210669 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.210854 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:33.211246 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.212048 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.213736 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.214247 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.214347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.214488 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.214764 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.214865 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":55,"threads_started":1}
I20260620 08:55:33.215359 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.209257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42431 (local address 127.0.0.1:33832)
0620 08:55:33.210277 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.210290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:33.210386 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:33.211014 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:33.211018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:33.211035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:33.211124 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.211130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.212162 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.212165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.213567 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.213578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.214352 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.214356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.214539 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.214561 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:33.214656 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 08:55:33.215244 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":564,"thread_start_us":85,"threads_started":1}
I20260620 08:55:33.216181 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.209211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46391 (local address 127.0.0.1:38124)
0620 08:55:33.209849 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.209899 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:33.210016 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:33.210856 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:33.210861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:33.210882 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:33.211025 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.211032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.212490 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.212494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.213579 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.213585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.215830 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.215834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.215916 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.215938 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:33.216007 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:55:33.216068 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":92,"threads_started":1}
I20260620 08:55:33.216552 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.209979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.0.98.2:46391)
0620 08:55:33.210369 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.210379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.210397 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:55:33.210466 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.210471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.210669 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.210854 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:33.211160 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.212355 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.214544 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.215706 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.215804 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.215926 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.216328 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.216421 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":115,"threads_started":1}
I20260620 08:55:33.217867 10768 tablet_service.cc:1467] Processing CreateTablet for tablet 8c34559344bd4d3faae9130c7affa24e (DEFAULT_TABLE table=test-table [id=cfbc5264e4a94c5993cbd7371cd310bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:33.217866 11036 tablet_service.cc:1467] Processing CreateTablet for tablet 8c34559344bd4d3faae9130c7affa24e (DEFAULT_TABLE table=test-table [id=cfbc5264e4a94c5993cbd7371cd310bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:33.218755 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c34559344bd4d3faae9130c7affa24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:33.218761 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c34559344bd4d3faae9130c7affa24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:33.223678 10902 tablet_service.cc:1467] Processing CreateTablet for tablet 8c34559344bd4d3faae9130c7affa24e (DEFAULT_TABLE table=test-table [id=cfbc5264e4a94c5993cbd7371cd310bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:55:33.224833 10902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c34559344bd4d3faae9130c7affa24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:55:33.225046 11119 tablet_bootstrap.cc:492] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: Bootstrap starting.
I20260620 08:55:33.226908 11120 tablet_bootstrap.cc:492] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Bootstrap starting.
I20260620 08:55:33.227260 11119 tablet_bootstrap.cc:654] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:33.228724 11119 log.cc:824] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:33.229063 11120 tablet_bootstrap.cc:654] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:33.230063 11120 log.cc:824] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:33.230805 11119 tablet_bootstrap.cc:492] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: No bootstrap required, opened a new log
I20260620 08:55:33.231011 11119 ts_tablet_manager.cc:1397] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 08:55:33.231731 11120 tablet_bootstrap.cc:492] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: No bootstrap required, opened a new log
I20260620 08:55:33.231881 11123 tablet_bootstrap.cc:492] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Bootstrap starting.
I20260620 08:55:33.231909 11120 ts_tablet_manager.cc:1397] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260620 08:55:33.234678 11123 tablet_bootstrap.cc:654] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Neither blocks nor log segments found. Creating new log.
I20260620 08:55:33.234830 11119 raft_consensus.cc:348] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.235051 11119 raft_consensus.cc:374] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:33.235101 11119 raft_consensus.cc:729] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fccf245a21ff4b92a8ac2d8b4ae7be79, State: Initialized, Role: FOLLOWER
I20260620 08:55:33.235502 11123 log.cc:824] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:55:33.235496 11119 consensus_queue.cc:260] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.235559 11120 raft_consensus.cc:348] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.235826 11120 raft_consensus.cc:374] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:33.235884 11120 raft_consensus.cc:729] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629f4911de294559af1e701ea6b72159, State: Initialized, Role: FOLLOWER
I20260620 08:55:33.236264 10834 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260620 08:55:33.236457 11119 ts_tablet_manager.cc:1428] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 08:55:33.236835 11120 consensus_queue.cc:260] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.237560 11123 tablet_bootstrap.cc:492] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: No bootstrap required, opened a new log
I20260620 08:55:33.237579 11102 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260620 08:55:33.237619 11120 ts_tablet_manager.cc:1428] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:55:33.237754 11123 ts_tablet_manager.cc:1397] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 08:55:33.241386 11123 raft_consensus.cc:348] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.241652 11123 raft_consensus.cc:374] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:55:33.241703 11123 raft_consensus.cc:729] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d7513239e2461e9e2b34ff14882d8a, State: Initialized, Role: FOLLOWER
I20260620 08:55:33.242162 11123 consensus_queue.cc:260] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.242715 10968 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260620 08:55:33.242957 11123 ts_tablet_manager.cc:1428] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260620 08:55:33.409933 11103 tablet.cc:2369] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:55:33.434620 10969 tablet.cc:2369] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:55:33.475077 10835 tablet.cc:2369] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:55:33.632705 11126 raft_consensus.cc:482] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:55:33.632925 11126 raft_consensus.cc:504] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.634186 11126 leader_election.cc:288] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d7513239e2461e9e2b34ff14882d8a (127.0.98.2:46391), fccf245a21ff4b92a8ac2d8b4ae7be79 (127.0.98.1:42431)
I20260620 08:55:33.637887 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.634617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49453 (local address 127.0.98.2:46391)
0620 08:55:33.634734 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.634738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.634768 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:55:33.634821 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.634823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.634872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.634957 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:33.635263 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.635837 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.636983 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.637357 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.637436 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.637503 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.637687 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.637765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:55:33.637926 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.634579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52843 (local address 127.0.98.1:42431)
0620 08:55:33.634697 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.634702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.634935 (+ 233us) server_negotiation.cc:407] Connection header received
0620 08:55:33.634995 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.634998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.635049 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.635181 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:55:33.635364 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.635893 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.637011 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.637390 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.637486 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.637545 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.637702 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.637828 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:55:33.637934 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.634496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42431 (local address 127.0.98.3:52843)
0620 08:55:33.634821 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.634832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:33.634925 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:33.635142 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:33.635145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:33.635158 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:33.635269 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.635276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.635994 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.635996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.636840 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.636849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.637503 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.637507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.637594 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.637611 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 08:55:33.637677 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:55:33.637749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":26,"thread_start_us":84,"threads_started":1}
I20260620 08:55:33.637908 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.634399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46391 (local address 127.0.98.3:49453)
0620 08:55:33.634524 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.634539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:55:33.634634 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:55:33.634991 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:55:33.634995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:55:33.635020 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:55:33.635148 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.635155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.635933 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.635936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.636851 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:55:33.636860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.637470 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.637474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:55:33.637572 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.637594 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:55:33.637655 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:55:33.637728 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":24}
I20260620 08:55:33.638602 10788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c34559344bd4d3faae9130c7affa24e" candidate_uuid: "629f4911de294559af1e701ea6b72159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" is_pre_election: true
I20260620 08:55:33.638670 10922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c34559344bd4d3faae9130c7affa24e" candidate_uuid: "629f4911de294559af1e701ea6b72159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d7513239e2461e9e2b34ff14882d8a" is_pre_election: true
I20260620 08:55:33.638890 10788 raft_consensus.cc:2454] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 629f4911de294559af1e701ea6b72159 in term 0.
I20260620 08:55:33.638911 10922 raft_consensus.cc:2454] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 629f4911de294559af1e701ea6b72159 in term 0.
I20260620 08:55:33.639322 10991 leader_election.cc:302] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 629f4911de294559af1e701ea6b72159, d7d7513239e2461e9e2b34ff14882d8a, fccf245a21ff4b92a8ac2d8b4ae7be79; no voters:
I20260620 08:55:33.639570 11126 raft_consensus.cc:2781] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:55:33.639675 11126 raft_consensus.cc:482] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:55:33.639725 11126 raft_consensus.cc:3037] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:33.640683 11126 raft_consensus.cc:504] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.641001 11126 leader_election.cc:288] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [CANDIDATE]: Term 1 election: Requested vote from peers d7d7513239e2461e9e2b34ff14882d8a (127.0.98.2:46391), fccf245a21ff4b92a8ac2d8b4ae7be79 (127.0.98.1:42431)
I20260620 08:55:33.641386 10788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c34559344bd4d3faae9130c7affa24e" candidate_uuid: "629f4911de294559af1e701ea6b72159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79"
I20260620 08:55:33.641422 10922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c34559344bd4d3faae9130c7affa24e" candidate_uuid: "629f4911de294559af1e701ea6b72159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d7513239e2461e9e2b34ff14882d8a"
I20260620 08:55:33.641522 10788 raft_consensus.cc:3037] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:33.641541 10922 raft_consensus.cc:3037] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:55:33.642885 10788 raft_consensus.cc:2454] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 629f4911de294559af1e701ea6b72159 in term 1.
I20260620 08:55:33.642885 10922 raft_consensus.cc:2454] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 629f4911de294559af1e701ea6b72159 in term 1.
I20260620 08:55:33.643275 10992 leader_election.cc:302] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629f4911de294559af1e701ea6b72159, fccf245a21ff4b92a8ac2d8b4ae7be79; no voters:
I20260620 08:55:33.643441 11126 raft_consensus.cc:2781] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:55:33.643669 11126 raft_consensus.cc:686] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 1 LEADER]: Becoming Leader. State: Replica: 629f4911de294559af1e701ea6b72159, State: Running, Role: LEADER
I20260620 08:55:33.643985 11126 consensus_queue.cc:237] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260620 08:55:33.647362 10641 catalog_manager.cc:5495] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 629f4911de294559af1e701ea6b72159 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "629f4911de294559af1e701ea6b72159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "629f4911de294559af1e701ea6b72159" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } health_report { overall_health: HEALTHY } } }
I20260620 08:55:33.668188 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.664585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.0.98.3:38839)
0620 08:55:33.664722 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:55:33.664726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:55:33.664741 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:55:33.664871 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:55:33.664873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:55:33.664928 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:55:33.665014 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:55:33.665265 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.665751 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.666559 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:55:33.666768 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:55:33.666854 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:55:33.667133 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:55:33.667560 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:55:33.667693 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 08:55:33.667752 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 08:55:33.667808 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:55:33.674333 10922 raft_consensus.cc:1261] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 1 FOLLOWER]: Refusing update from remote peer 629f4911de294559af1e701ea6b72159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:33.674336 10788 raft_consensus.cc:1261] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79 [term 1 FOLLOWER]: Refusing update from remote peer 629f4911de294559af1e701ea6b72159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:55:33.675482 11129 consensus_queue.cc:1035] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d7513239e2461e9e2b34ff14882d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:33.675791 11126 consensus_queue.cc:1035] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fccf245a21ff4b92a8ac2d8b4ae7be79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:55:33.685945 11134 mvcc.cc:205] Tried to move back new op lower bound from 7298849725121908736 to 7298849725008113664. Current Snapshot: MvccSnapshot[applied={T|T < 7298849725121908736}]
I20260620 08:55:33.691783 10641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:33.691937 10641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:55:33.694682 10641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 221aa5b2ddae4814b4f4725c5c7973db: Sending DeleteTablet for 3 replicas of tablet 8c34559344bd4d3faae9130c7affa24e
I20260620 08:55:33.695441 10902 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c34559344bd4d3faae9130c7affa24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260620 08:55:33.695447 10768 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c34559344bd4d3faae9130c7affa24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33832
I20260620 08:55:33.695999 11142 tablet_replica.cc:330] T 8c34559344bd4d3faae9130c7affa24e P fccf245a21ff4b92a8ac2d8b4ae7be79: stopping tablet replica
I20260620 08:55:33.696303 11141 tablet_replica.cc:330] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: stopping tablet replica
I20260620 08:55:33.696277 11036 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c34559344bd4d3faae9130c7affa24e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46052
I20260620 08:55:33.696621 11141 raft_consensus.cc:2229] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:55:33.696686 11143 tablet_replica.cc:330] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: stopping tablet replica
I20260620 08:55:33.696762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10705
I20260620 08:55:33.697309 11143 raft_consensus.cc:2229] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:55:33.697721 11143 raft_consensus.cc:2258] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:33.698310 11141 raft_consensus.cc:2258] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:55:33.699061 11143 ts_tablet_manager.cc:1905] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:33.699685 11141 ts_tablet_manager.cc:1905] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:55:33.701749 11143 ts_tablet_manager.cc:1918] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:33.701860 11143 log.cc:1196] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/8c34559344bd4d3faae9130c7affa24e
I20260620 08:55:33.702152 11143 ts_tablet_manager.cc:1939] T 8c34559344bd4d3faae9130c7affa24e P 629f4911de294559af1e701ea6b72159: Deleting consensus metadata
I20260620 08:55:33.702136 11141 ts_tablet_manager.cc:1918] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:55:33.702232 11141 log.cc:1196] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/8c34559344bd4d3faae9130c7affa24e
I20260620 08:55:33.702486 11141 ts_tablet_manager.cc:1939] T 8c34559344bd4d3faae9130c7affa24e P d7d7513239e2461e9e2b34ff14882d8a: Deleting consensus metadata
I20260620 08:55:33.703430 10628 catalog_manager.cc:4841] TS 629f4911de294559af1e701ea6b72159 (127.0.98.3:38839): tablet 8c34559344bd4d3faae9130c7affa24e (table test-table [id=cfbc5264e4a94c5993cbd7371cd310bc]) successfully deleted
I20260620 08:55:33.703779 10629 catalog_manager.cc:4841] TS d7d7513239e2461e9e2b34ff14882d8a (127.0.98.2:46391): tablet 8c34559344bd4d3faae9130c7affa24e (table test-table [id=cfbc5264e4a94c5993cbd7371cd310bc]) successfully deleted
W20260620 08:55:33.706904 10630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42431 (error 108)
I20260620 08:55:33.707083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10839
I20260620 08:55:33.712474 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:55:33.712170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42431 (local address 127.0.0.1:33846)
0620 08:55:33.712312 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:55:33.712383 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 08:55:33.712743 10630 catalog_manager.cc:4584] TS fccf245a21ff4b92a8ac2d8b4ae7be79 (127.0.98.1:42431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c34559344bd4d3faae9130c7affa24e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42431: connect: Connection refused (error 111)
I20260620 08:55:33.713012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10973
I20260620 08:55:33.717896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 10611
2026-06-20T08:55:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2538 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:55:33 dist-test-slave-fb1v krb5kdc[11147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:55:33 dist-test-slave-fb1v krb5kdc[11147](info): set up 2 sockets
Jun 20 08:55:33 dist-test-slave-fb1v krb5kdc[11147](info): commencing operation
krb5kdc: starting...
W20260620 08:55:34.684366 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:55:34 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:55:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 08:55:34.848748 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9wbfbl/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/postgres -l logfile start
2026-06-20 08:55:37.417 UTC [11177] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 08:55:37.417 UTC [11177] LOG: listening on IPv4 address "127.0.98.20", port 51249
2026-06-20 08:55:37.420 UTC [11177] LOG: listening on Unix socket "/tmp/.s.PGSQL.51249"
2026-06-20 08:55:37.447 UTC [11180] LOG: database system was shut down at 2026-06-20 08:55:35 UTC
2026-06-20 08:55:37.451 UTC [11177] LOG: database system is ready to accept connections
I20260620 08:55:38.324144 392 mini_postgres.cc:96] Postgres bound to 51249
2026-06-20 08:55:38.330 UTC [11188] FATAL: database "slave" does not exist
127.0.98.20:51249 - accepting connections
I20260620 08:55:38.331133 392 mini_ranger.cc:161] Starting Ranger...
I20260620 08:55:38.346683 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 08:55:38.807183 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 08:55:38.807279 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-admin
2026-06-20 08:55:39,065 [I] DB FLAVOR :POSTGRES
2026-06-20 08:55:39,066 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:55:39,066 [I] Checking connection..
2026-06-20 08:55:39,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 08:55:39,326 [I] Checking connection passed.
2026-06-20 08:55:39,326 [I] --------- Verifying version history table ---------
2026-06-20 08:55:39,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:55:39,566 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:55:39,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:55:39,795 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:55:39,795 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 08:55:39,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 08:55:40.057 UTC [11252] WARNING: there is no transaction in progress
2026-06-20 08:55:40,062 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 08:55:40,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:55:40,298 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 08:55:40,298 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:55:40,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 08:55:40,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 08:55:40,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fb1v.c.gcp-upstream.internal','N') ;"
2026-06-20 08:55:40,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:55:41,243 [I] Table x_portal_user does not exist in database ranger
2026-06-20 08:55:41,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:55:41,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 08:55:41,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:55:41,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:55:42,005 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 08:55:42,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 08:55:42.953 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:42.970 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:42.987 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.004 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.024 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.041 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.094 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.100 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.114 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.125 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.135 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.149 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.159 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.166 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.172 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43.178 UTC [11360] WARNING: there is no transaction in progress
2026-06-20 08:55:43,827 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 08:55:43,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 08:55:44,055 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 08:55:44,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:55:44,279 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 08:55:44,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:55:44,534 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 08:55:44,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:55:44,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:55:45,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fb1v.c.gcp-upstream.internal';"
2026-06-20 08:55:45,277 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 08:55:45,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 08:55:45,525 [I] DB_PATCHES have already been applied
I20260620 08:55:45.531580 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9wbfbl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 08:55:45.531682 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 08:55:45.535027 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39589
Jun 20, 2026 8:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:55:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:55:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 08:55:46.526091 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.195s user 0.000s sys 0.009s
Jun 20, 2026 8:56:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:56:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:56:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:56:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:56:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 08:56:07.272807 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 08:56:07 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945767, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 08:56:07.358491 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 08:56:07.367718 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 08:56:08.239921 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 08:56:08.240000 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms
I20260620 08:56:08.240067 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51249
2026-06-20 08:56:08,408 [I] DB FLAVOR :POSTGRES
2026-06-20 08:56:08,409 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:56:08,409 [I] Checking connection
2026-06-20 08:56:08,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 08:56:08,655 [I] connection success
2026-06-20 08:56:08,656 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 08:56:08,656 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 08:56:08,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 08:56:08,903 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 08:56:08,904 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:56:08,904 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 08:56:08,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 08:56:09,167 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 08:56:09.675222 392 mini_ranger_kms.cc:310] Created kms service
I20260620 08:56:09.893013 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 08:56:09.938905 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 08:56:09 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945767, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:10.330124 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 08:56:10.330273 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 08:56:10.330293 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 08:56:10.333930 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11540
I20260620 08:56:10.334065 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36333
Jun 20, 2026 8:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 08:56:12.375245 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.017s user 0.010s sys 0.144s
I20260620 08:56:12.375375 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 08:56:12.375474 392 mini_ranger_kms.cc:401] 127.0.98.20:36333/kms/v1/keys
Jun 20 08:56:12 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781945772, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 08:56:13 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945772, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 08:56:13 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945772, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 08:56:15 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:56:15.683446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46299
--ranger_config_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36333/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:56:15.794960 11591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:56:15.795220 11591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:56:15.795258 11591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:56:15.799273 11591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:56:15.799326 11591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:56:15.799343 11591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:56:15.799358 11591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:56:15.799372 11591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:56:15.805085 11591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36333/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46299
--ranger_config_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:56:15.806149 11591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:56:15.807155 11591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:56:15.815918 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:15.815927 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:15.816030 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:56:15.817013 11591 server_base.cc:1034] running on GCE node
I20260620 08:56:15.817647 11591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:56:15.819098 11591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:56:15.820300 11591 hybrid_clock.cc:648] HybridClock initialized: now 1781945775820276 us; error 33 us; skew 500 ppm
Jun 20 08:56:15 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:56:15.827045 11591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:56:15.828401 11591 webserver.cc:460] Webserver started at http://127.0.98.62:36723/ using document root <none> and password file <none>
I20260620 08:56:15.829155 11591 fs_manager.cc:359] Metadata directory not provided
I20260620 08:56:15.829243 11591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:56:15.829548 11591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:56:15 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:15.889191 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "cfddab11ccad49b9b3eb58a888ed8b46"
format_stamp: "Formatted at 2026-06-20 08:56:15 on dist-test-slave-fb1v"
server_key: "fb81fd04657d6e5545f10718d0403382"
server_key_iv: "10e44cf4e5d0a9ef2c1962b0904aa551"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:15.889850 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "cfddab11ccad49b9b3eb58a888ed8b46"
format_stamp: "Formatted at 2026-06-20 08:56:15 on dist-test-slave-fb1v"
server_key: "fb81fd04657d6e5545f10718d0403382"
server_key_iv: "10e44cf4e5d0a9ef2c1962b0904aa551"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:15.893667 11591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:56:15.912801 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:15.914413 11591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 08:56:15.914579 11591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "cfddab11ccad49b9b3eb58a888ed8b46"
format_stamp: "Formatted at 2026-06-20 08:56:15 on dist-test-slave-fb1v"
server_key: "fb81fd04657d6e5545f10718d0403382"
server_key_iv: "10e44cf4e5d0a9ef2c1962b0904aa551"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:15.914728 11591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:56:15.935561 11591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:56:15.939000 11591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:56:15.939301 11591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:56:15.946183 11591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46299
I20260620 08:56:15.946237 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46299 every 8 connection(s)
I20260620 08:56:15.947103 11591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:56:15.949842 11659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:56:15.951146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 11591
I20260620 08:56:15.951267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:56:15.951563 392 external_mini_cluster.cc:1428] Setting key d1abd72e4f57447f6fdb2d32fa6a19a8
I20260620 08:56:15.955753 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46: Bootstrap starting.
I20260620 08:56:15.958274 11659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46: Neither blocks nor log segments found. Creating new log.
I20260620 08:56:15.959115 11659 log.cc:824] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:56:15 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:56:15.961169 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46: No bootstrap required, opened a new log
I20260620 08:56:15.964802 11659 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46299 } }
I20260620 08:56:15.965013 11659 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:56:15.965066 11659 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfddab11ccad49b9b3eb58a888ed8b46, State: Initialized, Role: FOLLOWER
I20260620 08:56:15.965197 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:15.953098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.98.62:46299)
0620 08:56:15.953523 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:15.953537 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:15.953576 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:56:15.954256 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:15.954273 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:15.954594 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:15.954900 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:56:15.955688 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:15.956673 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:15.957491 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:15.957939 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:15.958098 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:15.960720 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:56:15.960726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:56:15.960739 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:56:15.960764 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:56:15.962246 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:15.962673 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:15.962677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:15.962681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:15.962780 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:15.963067 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:15.963070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:15.963072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:15.963511 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:56:15.963713 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:15.963947 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 08:56:15.964225 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260620 08:56:15.966354 11659 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46299 } }
I20260620 08:56:15.966490 11659 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:56:15.966562 11659 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:56:15.966652 11659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:56:15.967594 11659 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46299 } }
I20260620 08:56:15.967927 11659 leader_election.cc:302] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfddab11ccad49b9b3eb58a888ed8b46; no voters:
I20260620 08:56:15.968215 11659 leader_election.cc:288] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:56:15.968322 11664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:56:15.968555 11664 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [term 1 LEADER]: Becoming Leader. State: Replica: cfddab11ccad49b9b3eb58a888ed8b46, State: Running, Role: LEADER
I20260620 08:56:15.968858 11664 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46299 } }
I20260620 08:56:15.969249 11659 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:56:15.970330 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfddab11ccad49b9b3eb58a888ed8b46. Latest consensus state: current_term: 1 leader_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46299 } } }
I20260620 08:56:15.970317 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddab11ccad49b9b3eb58a888ed8b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46299 } } }
I20260620 08:56:15.970533 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [sys.catalog]: This master's current role is: LEADER
I20260620 08:56:15.970474 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46 [sys.catalog]: This master's current role is: LEADER
I20260620 08:56:15.971202 11659 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:56:15.971380 11675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:56:15.976434 11675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:56:15.983752 11675 catalog_manager.cc:1347] Generated new cluster ID: 5d7b456f7e434dfb8665a1fc32891ade
I20260620 08:56:15.983819 11675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:56:16.004242 11675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:56:16.005617 11675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:56:16.019821 11675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46: Generated new TSK 0
I20260620 08:56:16.020821 11675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:56:16.684913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46299
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:56:16.836226 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:56:16.836584 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:56:16.836648 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:56:16.842808 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:56:16.842897 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:56:16.842985 11704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:56:16.850504 11704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:56:16.851717 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:56:16.852874 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:56:16.862522 11714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:16.862644 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:16.862534 11711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:56:16.864027 11704 server_base.cc:1034] running on GCE node
I20260620 08:56:16.864456 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:56:16.865119 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:56:16.880604 11704 hybrid_clock.cc:648] HybridClock initialized: now 1781945776880497 us; error 117 us; skew 500 ppm
Jun 20 08:56:16 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:56:16.884358 11704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:56:16.885313 11704 webserver.cc:460] Webserver started at http://127.0.98.1:46317/ using document root <none> and password file <none>
I20260620 08:56:16.885867 11704 fs_manager.cc:359] Metadata directory not provided
I20260620 08:56:16.885931 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:56:16.886132 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:56:16 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:16.903784 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307"
format_stamp: "Formatted at 2026-06-20 08:56:16 on dist-test-slave-fb1v"
server_key: "1299260c7c8319bbc6b1158ffbd770c8"
server_key_iv: "540bf23c2564b4277979c9510f0d4565"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:16.904671 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307"
format_stamp: "Formatted at 2026-06-20 08:56:16 on dist-test-slave-fb1v"
server_key: "1299260c7c8319bbc6b1158ffbd770c8"
server_key_iv: "540bf23c2564b4277979c9510f0d4565"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:16.913898 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260620 08:56:16.935420 11723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:16.937227 11704 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 08:56:16.937361 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307"
format_stamp: "Formatted at 2026-06-20 08:56:16 on dist-test-slave-fb1v"
server_key: "1299260c7c8319bbc6b1158ffbd770c8"
server_key_iv: "540bf23c2564b4277979c9510f0d4565"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:16.937508 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:56:16.969794 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:56:16.974092 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:56:16.974306 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:56:16.975095 11704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:56:16.976248 11704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:56:16.976343 11704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:16.976420 11704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:56:16.976486 11704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:16.988281 11704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45689
I20260620 08:56:16.989174 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:56:16.989910 11836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45689 every 8 connection(s)
Jun 20 08:56:16 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781945776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:56:16.995514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 11704
I20260620 08:56:16.995666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:56:16.995962 392 external_mini_cluster.cc:1428] Setting key 38b30c2656a93391ec9b3fa5d1fd5ae2
Jun 20 08:56:16 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:56:17.007402 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:16.991676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46299 (local address 127.0.98.1:44963)
0620 08:56:16.992298 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:16.992353 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:16.993448 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:16.994183 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:16.994193 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:16.994527 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:56:16.995214 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:16.995236 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:16.996449 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:16.996452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:16.998369 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:16.998380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:16.999176 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:16.999181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:16.999385 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.000126 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:56:17.000146 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:56:17.001837 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:56:17.004133 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:56:17.004139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:56:17.004156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:56:17.004483 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:56:17.004792 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:56:17.004795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:56:17.004797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:56:17.004970 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:56:17.005470 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:56:17.005477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:56:17.005811 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.006018 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.006368 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":163,"threads_started":1}
I20260620 08:56:17.008102 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:16.991343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44963 (local address 127.0.98.62:46299)
0620 08:56:16.991801 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:16.991810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:16.992478 (+ 668us) server_negotiation.cc:407] Connection header received
0620 08:56:16.993703 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:16.993707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:16.993779 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:16.993878 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:56:16.995391 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:16.996287 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:16.998517 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:16.998879 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:16.998961 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.002050 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:56:17.002054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:56:17.002057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:56:17.002084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:56:17.003990 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.004596 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.004599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.004601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.004696 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.005091 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.005094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.005096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.005326 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:56:17.005417 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.007822 (+ 2405us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.007931 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":193,"threads_started":1}
I20260620 08:56:17.009111 11837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46299
I20260620 08:56:17.009361 11837 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:56:17.009876 11837 heartbeater.cc:511] Master 127.0.98.62:46299 requested a full tablet report, sending...
I20260620 08:56:17.011643 11623 ts_manager.cc:194] Registered new tserver with Master: f4b0b0cf7d064d0bbe04ffd7b2792307 (127.0.98.1:45689)
I20260620 08:56:17.013178 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:56:17.026221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46299
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:56:17.148066 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:56:17.148337 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:56:17.148393 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:56:17.152448 11843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:56:17.152508 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:56:17.152567 11843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:56:17.158985 11843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:56:17.160063 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:56:17.161031 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:56:17.172704 11850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:17.173223 11849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:17.173401 11852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:56:17.174324 11843 server_base.cc:1034] running on GCE node
I20260620 08:56:17.174911 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:56:17.175702 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:56:17.178957 11843 hybrid_clock.cc:648] HybridClock initialized: now 1781945777178924 us; error 38 us; skew 500 ppm
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:56:17.181591 11843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:56:17.182575 11843 webserver.cc:460] Webserver started at http://127.0.98.2:46299/ using document root <none> and password file <none>
I20260620 08:56:17.183104 11843 fs_manager.cc:359] Metadata directory not provided
I20260620 08:56:17.183169 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:56:17.183398 11843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:17.197253 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "3cb6a103589e4511b343acfc6f17d449"
format_stamp: "Formatted at 2026-06-20 08:56:17 on dist-test-slave-fb1v"
server_key: "8df9d7c9a81e72914bd85a04859b5f6a"
server_key_iv: "76ae0ddb238b3cc98ac493964c857b52"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:17.198079 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3cb6a103589e4511b343acfc6f17d449"
format_stamp: "Formatted at 2026-06-20 08:56:17 on dist-test-slave-fb1v"
server_key: "8df9d7c9a81e72914bd85a04859b5f6a"
server_key_iv: "76ae0ddb238b3cc98ac493964c857b52"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:17.210752 11843 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260620 08:56:17.234131 11859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:17.235785 11843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 08:56:17.235918 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "3cb6a103589e4511b343acfc6f17d449"
format_stamp: "Formatted at 2026-06-20 08:56:17 on dist-test-slave-fb1v"
server_key: "8df9d7c9a81e72914bd85a04859b5f6a"
server_key_iv: "76ae0ddb238b3cc98ac493964c857b52"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:17.236053 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:56:17.268433 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:56:17.272178 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:56:17.272367 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:56:17.273124 11843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:56:17.274312 11843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:56:17.274376 11843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:17.274431 11843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:56:17.274459 11843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:17.284731 11843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36305
I20260620 08:56:17.284960 11972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36305 every 8 connection(s)
I20260620 08:56:17.285943 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:56:17.296249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 11843
I20260620 08:56:17.296386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:56:17.296697 392 external_mini_cluster.cc:1428] Setting key a7d3fde3823458bb61f2702eafb17540
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:56:17.303545 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.288755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39557 (local address 127.0.98.62:46299)
0620 08:56:17.289059 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.289063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.289769 (+ 706us) server_negotiation.cc:407] Connection header received
0620 08:56:17.290954 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.290958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.291015 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.291107 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:56:17.292527 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.293338 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.294980 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.295339 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.295452 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.298913 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:56:17.298918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:56:17.298920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:56:17.298952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:56:17.300822 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.301574 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.301578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.301580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.301681 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.302143 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.302146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.302148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.302372 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:56:17.303013 (+ 641us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.303181 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.303342 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 08:56:17.304697 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.289021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46299 (local address 127.0.98.2:39557)
0620 08:56:17.289598 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.289664 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.290694 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.291295 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.291305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.291695 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:56:17.292350 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.292371 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.293478 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.293482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.294844 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.294855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.295503 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.295507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.295739 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.296409 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:56:17.296430 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:56:17.298693 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:56:17.300987 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:56:17.300994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:56:17.301016 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:56:17.301442 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:56:17.301804 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:56:17.301808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:56:17.301810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:56:17.302017 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:56:17.302493 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:56:17.302501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:56:17.302882 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.303137 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.303511 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":150,"threads_started":1}
I20260620 08:56:17.306417 11973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46299
I20260620 08:56:17.306696 11973 heartbeater.cc:464] Registering TS with master...
I20260620 08:56:17.307300 11973 heartbeater.cc:511] Master 127.0.98.62:46299 requested a full tablet report, sending...
I20260620 08:56:17.308557 11623 ts_manager.cc:194] Registered new tserver with Master: 3cb6a103589e4511b343acfc6f17d449 (127.0.98.2:36305)
I20260620 08:56:17.309317 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39557
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:56:17.343569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46299
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:56:17.523231 11978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:56:17.523758 11978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:56:17.523872 11978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:56:17.530645 11978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:56:17.530969 11978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:56:17.531108 11978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:56:17.539521 11978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:56:17.542060 11978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:56:17.544044 11978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:56:17.557853 11978 server_base.cc:1034] running on GCE node
W20260620 08:56:17.558259 11984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:17.558548 11983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:56:17.559469 11986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:56:17.560086 11978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:56:17.560861 11978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:17.564108 11978 hybrid_clock.cc:648] HybridClock initialized: now 1781945777564026 us; error 86 us; skew 500 ppm
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:56:17.568049 11978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:56:17.569501 11978 webserver.cc:460] Webserver started at http://127.0.98.3:39129/ using document root <none> and password file <none>
I20260620 08:56:17.570287 11978 fs_manager.cc:359] Metadata directory not provided
I20260620 08:56:17.570430 11978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:56:17.570750 11978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:17.586124 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "7cb0896d706b406182c8e7e3257f4fdf"
format_stamp: "Formatted at 2026-06-20 08:56:17 on dist-test-slave-fb1v"
server_key: "63ee6a69a21b046f0989544ccae6d271"
server_key_iv: "7e4b510985073004b0ecf143570422fe"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:17.586943 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7cb0896d706b406182c8e7e3257f4fdf"
format_stamp: "Formatted at 2026-06-20 08:56:17 on dist-test-slave-fb1v"
server_key: "63ee6a69a21b046f0989544ccae6d271"
server_key_iv: "7e4b510985073004b0ecf143570422fe"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:17.601919 11978 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260620 08:56:17.630621 11993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:17.632633 11978 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.002s
I20260620 08:56:17.632786 11978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "7cb0896d706b406182c8e7e3257f4fdf"
format_stamp: "Formatted at 2026-06-20 08:56:17 on dist-test-slave-fb1v"
server_key: "63ee6a69a21b046f0989544ccae6d271"
server_key_iv: "7e4b510985073004b0ecf143570422fe"
server_key_version: "kuduclusterkey@0"
I20260620 08:56:17.632936 11978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:56:17.672837 11978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:56:17.680580 11978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:56:17.680858 11978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:56:17.681622 11978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:56:17.682713 11978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:56:17.682811 11978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:17.682905 11978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:56:17.682966 11978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:56:17.697618 11978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42951
I20260620 08:56:17.698817 11978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:56:17.699940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 11978
I20260620 08:56:17.700068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:56:17.700392 392 external_mini_cluster.cc:1428] Setting key 49c4404388312e4523a37e66e0ccf85b
I20260620 08:56:17.705264 12106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42951 every 8 connection(s)
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:56:17.724362 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.707775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48907 (local address 127.0.98.62:46299)
0620 08:56:17.707891 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.707895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.709259 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 08:56:17.710399 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.710402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.710451 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.710498 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:56:17.712626 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.713483 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.716399 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.716751 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.716818 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.719865 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:56:17.719869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:56:17.719872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:56:17.719898 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:56:17.721843 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.722484 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.722488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.722490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.722568 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.722921 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.722924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.722925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.723156 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:56:17.723229 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.723987 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.724116 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 08:56:17.727213 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.707441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46299 (local address 127.0.98.3:48907)
0620 08:56:17.709114 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.709163 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.710171 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.710907 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.710916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.711255 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:56:17.712337 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.712379 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.714138 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.714142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.716269 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.716282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.717005 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.717009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.717214 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.717879 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:56:17.717899 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:56:17.719665 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:56:17.722020 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:56:17.722025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:56:17.722042 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:56:17.722374 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:56:17.722663 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:56:17.722666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:56:17.722668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:56:17.722826 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:56:17.723355 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:56:17.723362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:56:17.723714 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.724802 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.725625 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"thread_start_us":388,"threads_started":1}
I20260620 08:56:17.728580 12107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46299
I20260620 08:56:17.728812 12107 heartbeater.cc:464] Registering TS with master...
I20260620 08:56:17.729341 12107 heartbeater.cc:511] Master 127.0.98.62:46299 requested a full tablet report, sending...
I20260620 08:56:17.731985 11623 ts_manager.cc:194] Registered new tserver with Master: 7cb0896d706b406182c8e7e3257f4fdf (127.0.98.3:42951)
I20260620 08:56:17.732808 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48907
I20260620 08:56:17.739583 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:56:17.757328 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.747734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:46299)
0620 08:56:17.747847 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.747851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.748190 (+ 339us) server_negotiation.cc:407] Connection header received
0620 08:56:17.748376 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.748380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.748427 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.748475 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:56:17.749433 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.750260 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.751528 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.751903 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.751975 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.753805 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:56:17.753809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:56:17.753811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:56:17.753837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:56:17.755482 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.755973 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.755978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.755979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.756078 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:56:17.756367 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:56:17.756370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:56:17.756372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:56:17.756584 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:56:17.756687 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.757028 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.757159 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:56:17.761088 11623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:56:17.765395 11623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:56:17.792088 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.778120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42951 (local address 127.0.0.1:35436)
0620 08:56:17.781401 (+ 3281us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.781412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.781484 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.783719 (+ 2235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.783722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.783738 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:17.783812 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.783818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.785089 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.785092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.786997 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.787008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.791545 (+ 4537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.791548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.791681 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.791720 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.791813 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.791926 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3228,"thread_start_us":76,"threads_started":1}
I20260620 08:56:17.794243 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.779252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.98.3:42951)
0620 08:56:17.781285 (+ 2033us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.781294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.781556 (+ 262us) server_negotiation.cc:407] Connection header received
0620 08:56:17.781625 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.781630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.781806 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.781931 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:17.783921 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.784963 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.787858 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.788613 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.788706 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.793960 (+ 5254us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.794007 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.794112 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1945,"thread_start_us":103,"threads_started":1}
I20260620 08:56:17.792841 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.778003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36305 (local address 127.0.0.1:44808)
0620 08:56:17.780312 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.780358 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.780470 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.782712 (+ 2242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.782717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.782738 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:17.782861 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.782866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.789497 (+ 6631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.789500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.790562 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.790572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.792522 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.792526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.792608 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.792628 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.792670 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.792727 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2147,"thread_start_us":110,"threads_started":1}
I20260620 08:56:17.800251 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.779216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45689 (local address 127.0.0.1:41992)
0620 08:56:17.794827 (+ 15611us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.794840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.794972 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.795598 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.795601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.795634 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:17.795698 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.795702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.796960 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.796963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.798024 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.798032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.799910 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.799914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.799991 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.800009 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.800073 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.800132 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15558,"thread_start_us":77,"threads_started":1}
I20260620 08:56:17.800709 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.779349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.98.1:45689)
0620 08:56:17.780076 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.780084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.794907 (+ 14823us) server_negotiation.cc:407] Connection header received
0620 08:56:17.795110 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.795116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.795330 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.795444 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:17.795803 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.796833 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.798159 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.799636 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.799707 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.799832 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.800487 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.800586 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260620 08:56:17.804092 11771 tablet_service.cc:1467] Processing CreateTablet for tablet 65f91658803345f99858c6898f635231 (DEFAULT_TABLE table=test-table [id=715d05789ab5408b81cf688ecb5f8123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:56:17.805233 11771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f91658803345f99858c6898f635231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:56:17.818315 12039 tablet_service.cc:1467] Processing CreateTablet for tablet 65f91658803345f99858c6898f635231 (DEFAULT_TABLE table=test-table [id=715d05789ab5408b81cf688ecb5f8123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:56:17.819406 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f91658803345f99858c6898f635231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:56:17.828497 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.778913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.0.98.2:36305)
0620 08:56:17.780735 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.780748 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.780769 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:56:17.780835 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.780840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.781032 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.782994 (+ 1962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:17.783123 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.787733 (+ 4610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.790720 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.791438 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.792932 (+ 1494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.793090 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.793141 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.793229 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":105,"threads_started":1}
I20260620 08:56:17.830379 12126 tablet_bootstrap.cc:492] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Bootstrap starting.
I20260620 08:56:17.832245 11907 tablet_service.cc:1467] Processing CreateTablet for tablet 65f91658803345f99858c6898f635231 (DEFAULT_TABLE table=test-table [id=715d05789ab5408b81cf688ecb5f8123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:56:17.833429 11907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f91658803345f99858c6898f635231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:56:17.833698 12126 tablet_bootstrap.cc:654] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Neither blocks nor log segments found. Creating new log.
I20260620 08:56:17.834519 12126 log.cc:824] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Log is configured to *not* fsync() on all Append() calls
I20260620 08:56:17.830854 12125 tablet_bootstrap.cc:492] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: Bootstrap starting.
I20260620 08:56:17.838881 12125 tablet_bootstrap.cc:654] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: Neither blocks nor log segments found. Creating new log.
I20260620 08:56:17.840081 12125 log.cc:824] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: Log is configured to *not* fsync() on all Append() calls
I20260620 08:56:17.842569 12126 tablet_bootstrap.cc:492] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: No bootstrap required, opened a new log
I20260620 08:56:17.842854 12126 ts_tablet_manager.cc:1397] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260620 08:56:17.846763 12126 raft_consensus.cc:348] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.847177 12126 raft_consensus.cc:374] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:56:17.847265 12126 raft_consensus.cc:729] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb0896d706b406182c8e7e3257f4fdf, State: Initialized, Role: FOLLOWER
I20260620 08:56:17.847798 12126 consensus_queue.cc:260] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.849350 12126 ts_tablet_manager.cc:1428] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:56:17.850845 12125 tablet_bootstrap.cc:492] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: No bootstrap required, opened a new log
I20260620 08:56:17.851087 12125 ts_tablet_manager.cc:1397] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260620 08:56:17.851966 12129 tablet_bootstrap.cc:492] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Bootstrap starting.
I20260620 08:56:17.854465 12129 tablet_bootstrap.cc:654] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Neither blocks nor log segments found. Creating new log.
I20260620 08:56:17.855388 12129 log.cc:824] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Log is configured to *not* fsync() on all Append() calls
I20260620 08:56:17.856791 12125 raft_consensus.cc:348] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.857159 12125 raft_consensus.cc:374] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:56:17.857239 12125 raft_consensus.cc:729] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b0b0cf7d064d0bbe04ffd7b2792307, State: Initialized, Role: FOLLOWER
I20260620 08:56:17.857764 12125 consensus_queue.cc:260] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.859051 12125 ts_tablet_manager.cc:1428] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 08:56:17.859298 12107 heartbeater.cc:503] Master 127.0.98.62:46299 was elected leader, sending a full tablet report...
I20260620 08:56:17.860721 11837 heartbeater.cc:503] Master 127.0.98.62:46299 was elected leader, sending a full tablet report...
I20260620 08:56:17.863062 12129 tablet_bootstrap.cc:492] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: No bootstrap required, opened a new log
I20260620 08:56:17.863247 12129 ts_tablet_manager.cc:1397] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260620 08:56:17.867821 12129 raft_consensus.cc:348] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.868075 12129 raft_consensus.cc:374] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:56:17.868130 12129 raft_consensus.cc:729] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb6a103589e4511b343acfc6f17d449, State: Initialized, Role: FOLLOWER
I20260620 08:56:17.868700 12129 consensus_queue.cc:260] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.869907 12129 ts_tablet_manager.cc:1428] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 08:56:17.870275 11973 heartbeater.cc:503] Master 127.0.98.62:46299 was elected leader, sending a full tablet report...
W20260620 08:56:17.949553 11654 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.62:46299
W20260620 08:56:17.959163 12108 tablet.cc:2369] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 08:56:17 dist-test-slave-fb1v krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:17.981060 12130 raft_consensus.cc:482] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:56:17.981300 12130 raft_consensus.cc:504] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.982704 12130 leader_election.cc:288] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb6a103589e4511b343acfc6f17d449 (127.0.98.2:36305), f4b0b0cf7d064d0bbe04ffd7b2792307 (127.0.98.1:45689)
I20260620 08:56:17.987567 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.982907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36305 (local address 127.0.98.3:59643)
0620 08:56:17.983186 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.983201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.983319 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.983878 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.983881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.983904 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:17.984025 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.984031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.985034 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.985038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.986226 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.986237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.987189 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.987192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.987287 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.987309 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.987369 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.987440 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260620 08:56:17.988451 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.983486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59643 (local address 127.0.98.2:36305)
0620 08:56:17.983641 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.983645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.983660 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:56:17.983712 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.983715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.983763 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.983844 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:17.984172 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.984909 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.986378 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.986938 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.987026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.987088 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.988268 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.988344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:56:17.989060 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f91658803345f99858c6898f635231" candidate_uuid: "7cb0896d706b406182c8e7e3257f4fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb6a103589e4511b343acfc6f17d449" is_pre_election: true
I20260620 08:56:17.989395 11927 raft_consensus.cc:2454] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb0896d706b406182c8e7e3257f4fdf in term 0.
I20260620 08:56:17.989969 11996 leader_election.cc:302] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb6a103589e4511b343acfc6f17d449, 7cb0896d706b406182c8e7e3257f4fdf; no voters:
I20260620 08:56:17.990392 12130 raft_consensus.cc:2781] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:56:17.990479 12130 raft_consensus.cc:482] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:56:17.990518 12130 raft_consensus.cc:3037] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:56:17.991945 12130 raft_consensus.cc:504] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.992084 12131 raft_consensus.cc:482] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:56:17.992218 12131 raft_consensus.cc:504] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.992794 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f91658803345f99858c6898f635231" candidate_uuid: "7cb0896d706b406182c8e7e3257f4fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb6a103589e4511b343acfc6f17d449"
I20260620 08:56:17.992918 11927 raft_consensus.cc:3037] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:56:17.993283 12131 leader_election.cc:288] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb6a103589e4511b343acfc6f17d449 (127.0.98.2:36305), 7cb0896d706b406182c8e7e3257f4fdf (127.0.98.3:42951)
I20260620 08:56:17.994184 11927 raft_consensus.cc:2454] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb0896d706b406182c8e7e3257f4fdf in term 1.
I20260620 08:56:17.994553 11996 leader_election.cc:302] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb6a103589e4511b343acfc6f17d449, 7cb0896d706b406182c8e7e3257f4fdf; no voters:
I20260620 08:56:17.994560 12130 leader_election.cc:288] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [CANDIDATE]: Term 1 election: Requested vote from peers 3cb6a103589e4511b343acfc6f17d449 (127.0.98.2:36305), f4b0b0cf7d064d0bbe04ffd7b2792307 (127.0.98.1:45689)
I20260620 08:56:17.994793 12130 raft_consensus.cc:2781] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:56:17.994904 12130 raft_consensus.cc:686] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 1 LEADER]: Becoming Leader. State: Replica: 7cb0896d706b406182c8e7e3257f4fdf, State: Running, Role: LEADER
I20260620 08:56:17.995257 12130 consensus_queue.cc:237] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } }
I20260620 08:56:17.995473 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.983505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37623 (local address 127.0.98.1:45689)
0620 08:56:17.983645 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.983649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.987996 (+ 4347us) server_negotiation.cc:407] Connection header received
0620 08:56:17.988050 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.988053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.988100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.988170 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:17.988953 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.989746 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.991106 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.991721 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.991813 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.991874 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:17.995297 (+ 3423us) server_negotiation.cc:299] Negotiation successful
0620 08:56:17.995372 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 08:56:17.995743 11838 tablet.cc:2369] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:56:17.995711 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.983081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45689 (local address 127.0.98.3:37623)
0620 08:56:17.987750 (+ 4669us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.987763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.987836 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.988740 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.988743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.988762 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:17.988834 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.988839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.989882 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.989886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.990963 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.990974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.994984 (+ 4010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.994988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.995093 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:17.995116 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:56:17.995179 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:56:17.995245 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4617,"thread_start_us":127,"threads_started":1}
I20260620 08:56:17.997960 11791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f91658803345f99858c6898f635231" candidate_uuid: "7cb0896d706b406182c8e7e3257f4fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" is_pre_election: true
I20260620 08:56:17.998193 11791 raft_consensus.cc:2454] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb0896d706b406182c8e7e3257f4fdf in term 0.
I20260620 08:56:17.998272 11790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f91658803345f99858c6898f635231" candidate_uuid: "7cb0896d706b406182c8e7e3257f4fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307"
I20260620 08:56:17.998382 11790 raft_consensus.cc:3037] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:56:17.999691 11790 raft_consensus.cc:2454] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb0896d706b406182c8e7e3257f4fdf in term 1.
I20260620 08:56:18.000900 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.994292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57299 (local address 127.0.98.2:36305)
0620 08:56:17.994934 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.994938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:17.995949 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 08:56:17.996087 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:17.996089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:17.996136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:17.996211 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:17.996499 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.997219 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.999705 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.000301 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.000366 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:18.000421 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:18.000719 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 08:56:18.000790 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574}
I20260620 08:56:18.001361 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.993561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36305 (local address 127.0.98.1:57299)
0620 08:56:17.995873 (+ 2312us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:17.995887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:17.995981 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:17.996222 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:17.996225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:17.996242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:17.996362 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.996369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:17.997349 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:17.997352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:17.999535 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:17.999545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.000503 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.000507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:18.000600 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:18.000622 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:56:18.001161 (+ 539us) client_negotiation.cc:240] Negotiation successful
0620 08:56:18.001236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2249,"thread_start_us":103,"threads_started":1}
I20260620 08:56:18.001104 11622 catalog_manager.cc:5495] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cb0896d706b406182c8e7e3257f4fdf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7cb0896d706b406182c8e7e3257f4fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cb0896d706b406182c8e7e3257f4fdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 } health_report { overall_health: UNKNOWN } } }
I20260620 08:56:18.001864 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f91658803345f99858c6898f635231" candidate_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb6a103589e4511b343acfc6f17d449" is_pre_election: true
I20260620 08:56:18.002031 11927 raft_consensus.cc:2379] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4b0b0cf7d064d0bbe04ffd7b2792307 in current term 1: Already voted for candidate 7cb0896d706b406182c8e7e3257f4fdf in this term.
I20260620 08:56:18.007027 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.994458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49383 (local address 127.0.98.3:42951)
0620 08:56:17.994845 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:17.994849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:18.001919 (+ 7070us) server_negotiation.cc:407] Connection header received
0620 08:56:18.002541 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:18.002544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:18.002588 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:18.002636 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:56:18.003504 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.004202 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.005914 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.006501 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.006603 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:18.006661 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:18.006837 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 08:56:18.006915 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260620 08:56:18.007419 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:17.993976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42951 (local address 127.0.98.1:49383)
0620 08:56:18.001864 (+ 7888us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:18.001876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:18.001970 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:18.003316 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:56:18.003319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:56:18.003336 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:56:18.003404 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:18.003409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.004319 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.004322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:18.005777 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:56:18.005788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.006618 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.006622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:56:18.006702 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:18.006725 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:56:18.007218 (+ 493us) client_negotiation.cc:240] Negotiation successful
0620 08:56:18.007306 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7835,"thread_start_us":96,"threads_started":1}
I20260620 08:56:18.008100 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f91658803345f99858c6898f635231" candidate_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb0896d706b406182c8e7e3257f4fdf" is_pre_election: true
I20260620 08:56:18.008901 11727 leader_election.cc:302] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4b0b0cf7d064d0bbe04ffd7b2792307; no voters: 3cb6a103589e4511b343acfc6f17d449, 7cb0896d706b406182c8e7e3257f4fdf
I20260620 08:56:18.009402 12131 raft_consensus.cc:2726] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 08:56:18.020107 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:18.014229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.98.3:42951)
0620 08:56:18.014336 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:56:18.014340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:56:18.014351 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:56:18.014417 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:56:18.014419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:56:18.014474 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:56:18.014563 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:56:18.015027 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.015818 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.017899 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:56:18.018240 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:56:18.018335 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:56:18.018721 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:56:18.019260 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:56:18.019444 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 08:56:18.019894 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 08:56:18.019950 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:56:18.027388 11927 raft_consensus.cc:1261] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 1 FOLLOWER]: Refusing update from remote peer 7cb0896d706b406182c8e7e3257f4fdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:56:18.027436 11790 raft_consensus.cc:1261] T 65f91658803345f99858c6898f635231 P f4b0b0cf7d064d0bbe04ffd7b2792307 [term 1 FOLLOWER]: Refusing update from remote peer 7cb0896d706b406182c8e7e3257f4fdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:56:18.028080 12130 consensus_queue.cc:1035] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b0b0cf7d064d0bbe04ffd7b2792307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:56:18.028477 12130 consensus_queue.cc:1035] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb6a103589e4511b343acfc6f17d449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:56:18.037338 11974 tablet.cc:2369] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:56:18.038427 12141 mvcc.cc:205] Tried to move back new op lower bound from 7298849906792722432 to 7298849906672680960. Current Snapshot: MvccSnapshot[applied={T|T < 7298849906792722432}]
I20260620 08:56:18.046185 12142 mvcc.cc:205] Tried to move back new op lower bound from 7298849906792722432 to 7298849906672680960. Current Snapshot: MvccSnapshot[applied={T|T < 7298849906792722432}]
I20260620 08:56:18.048835 12144 mvcc.cc:205] Tried to move back new op lower bound from 7298849906792722432 to 7298849906672680960. Current Snapshot: MvccSnapshot[applied={T|T < 7298849906792722432}]
I20260620 08:56:18.053591 11622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:56:18.053769 11622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:56:18.056454 11622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfddab11ccad49b9b3eb58a888ed8b46: Sending DeleteTablet for 3 replicas of tablet 65f91658803345f99858c6898f635231
I20260620 08:56:18.057617 11907 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f91658803345f99858c6898f635231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:56:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44808
I20260620 08:56:18.058076 11771 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f91658803345f99858c6898f635231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:56:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41992
I20260620 08:56:18.059278 12039 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f91658803345f99858c6898f635231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:56:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35436
I20260620 08:56:18.061443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 11704
I20260620 08:56:18.061621 12154 tablet_replica.cc:330] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: stopping tablet replica
I20260620 08:56:18.061906 12154 raft_consensus.cc:2229] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:56:18.062294 12154 raft_consensus.cc:2258] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:56:18.066224 12152 tablet_replica.cc:330] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: stopping tablet replica
I20260620 08:56:18.069077 12154 ts_tablet_manager.cc:1905] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:56:18.070442 12152 raft_consensus.cc:2229] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:56:18.071943 12154 ts_tablet_manager.cc:1918] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:56:18.072121 12154 log.cc:1196] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/65f91658803345f99858c6898f635231
I20260620 08:56:18.072525 12154 ts_tablet_manager.cc:1939] T 65f91658803345f99858c6898f635231 P 7cb0896d706b406182c8e7e3257f4fdf: Deleting consensus metadata
I20260620 08:56:18.073884 11610 catalog_manager.cc:4841] TS 7cb0896d706b406182c8e7e3257f4fdf (127.0.98.3:42951): tablet 65f91658803345f99858c6898f635231 (table test-table [id=715d05789ab5408b81cf688ecb5f8123]) successfully deleted
I20260620 08:56:18.077559 12152 raft_consensus.cc:2258] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:56:18.081142 12152 ts_tablet_manager.cc:1905] T 65f91658803345f99858c6898f635231 P 3cb6a103589e4511b343acfc6f17d449: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:56:18.081497 11610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45689 (error 108)
I20260620 08:56:18.082847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 11843
I20260620 08:56:18.082914 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:56:18.082593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45689 (local address 127.0.0.1:42002)
0620 08:56:18.082732 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:18.082820 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 08:56:18.083806 11610 catalog_manager.cc:4584] TS f4b0b0cf7d064d0bbe04ffd7b2792307 (127.0.98.1:45689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65f91658803345f99858c6898f635231: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45689: connect: Connection refused (error 111)
W20260620 08:56:18.090267 12120 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:56:18.089668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36305 (local address 127.0.0.1:44822)
0620 08:56:18.089846 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:56:18.089860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:56:18.089956 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:56:18.090148 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 08:56:18.090893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 11978
I20260620 08:56:18.097712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 11591
I20260620 08:56:18.104724 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 08:56:18.439633 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 08:56:18.439767 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 08:56:18.775282 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 08:56:18.775 UTC [11177] LOG: received smart shutdown request
2026-06-20 08:56:18.779 UTC [11177] LOG: background worker "logical replication launcher" (PID 11186) exited with exit code 1
2026-06-20 08:56:18.779 UTC [11181] LOG: shutting down
2026-06-20 08:56:18.959 UTC [11177] LOG: database system is shut down
2026-06-20T08:56:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45345 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:56:19 dist-test-slave-fb1v krb5kdc[12164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:56:19 dist-test-slave-fb1v krb5kdc[12164](info): set up 2 sockets
Jun 20 08:56:19 dist-test-slave-fb1v krb5kdc[12164](info): commencing operation
krb5kdc: starting...
W20260620 08:56:20.035658 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:56:20 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:56:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 08:56:20.202098 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9wbfbl/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/postgres -l logfile start
2026-06-20 08:56:22.833 UTC [12194] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 08:56:22.833 UTC [12194] LOG: listening on IPv4 address "127.0.98.20", port 59931
2026-06-20 08:56:22.837 UTC [12194] LOG: listening on Unix socket "/tmp/.s.PGSQL.59931"
2026-06-20 08:56:22.865 UTC [12197] LOG: database system was shut down at 2026-06-20 08:56:21 UTC
2026-06-20 08:56:22.869 UTC [12194] LOG: database system is ready to accept connections
I20260620 08:56:23.747303 392 mini_postgres.cc:96] Postgres bound to 59931
2026-06-20 08:56:23.753 UTC [12205] FATAL: database "slave" does not exist
127.0.98.20:59931 - accepting connections
I20260620 08:56:23.753983 392 mini_ranger.cc:161] Starting Ranger...
I20260620 08:56:23.768260 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 08:56:24.243916 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 08:56:24.243999 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-admin
2026-06-20 08:56:24,517 [I] DB FLAVOR :POSTGRES
2026-06-20 08:56:24,518 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:56:24,518 [I] Checking connection..
2026-06-20 08:56:24,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 08:56:24,759 [I] Checking connection passed.
2026-06-20 08:56:24,760 [I] --------- Verifying version history table ---------
2026-06-20 08:56:24,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:56:25,007 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:56:25,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:56:25,266 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:56:25,267 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 08:56:25,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 08:56:25.500 UTC [12269] WARNING: there is no transaction in progress
2026-06-20 08:56:25,505 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 08:56:25,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:56:25,739 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 08:56:25,739 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:56:25,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 08:56:25,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 08:56:26,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fb1v.c.gcp-upstream.internal','N') ;"
2026-06-20 08:56:26,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:56:26,699 [I] Table x_portal_user does not exist in database ranger
2026-06-20 08:56:26,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:56:26,958 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 08:56:26,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:56:27,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:56:27,466 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 08:56:27,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 08:56:28.467 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.484 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.501 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.516 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.534 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.555 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.611 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.618 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.632 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.643 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.653 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.668 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.680 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.688 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.695 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:28.702 UTC [12377] WARNING: there is no transaction in progress
2026-06-20 08:56:29,331 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 08:56:29,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 08:56:29,568 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 08:56:29,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:56:29,806 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 08:56:29,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:56:30,064 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 08:56:30,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:56:30,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:56:30,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fb1v.c.gcp-upstream.internal';"
2026-06-20 08:56:30,805 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 08:56:30,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 08:56:31,035 [I] DB_PATCHES have already been applied
I20260620 08:56:31.041100 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9wbfbl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 08:56:31.041163 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 08:56:31.044289 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44261
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 08:56:32.041590 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.288s user 0.002s sys 0.006s
Jun 20, 2026 8:56:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:56:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:56:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:56:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:56:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 08:56:54.110592 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 08:56:54 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945814, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 08:56:54.186293 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 08:56:54.195650 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 08:56:55.031867 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 08:56:55.031940 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms
I20260620 08:56:55.031957 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59931
2026-06-20 08:56:55,183 [I] DB FLAVOR :POSTGRES
2026-06-20 08:56:55,183 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:56:55,183 [I] Checking connection
2026-06-20 08:56:55,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 08:56:55,419 [I] connection success
2026-06-20 08:56:55,419 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 08:56:55,419 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 08:56:55,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 08:56:55,642 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 08:56:55,642 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:56:55,642 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 08:56:55,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59931/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 08:56:55,910 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 08:56:56.337767 392 mini_ranger_kms.cc:310] Created kms service
I20260620 08:56:56.525484 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 08:56:56.569515 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 08:56:56 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945814, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:56:56.979347 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 08:56:56.979638 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task9wbfbl/build/debug/bin/postgresql.jar:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9wbfbl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 08:56:56.979664 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 08:56:56.983714 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12557
I20260620 08:56:56.983807 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59067
Jun 20, 2026 8:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:56:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:56:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:56:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:56:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 08:56:59.028990 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.843s user 0.022s sys 0.114s
I20260620 08:56:59.029090 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 08:56:59.029170 392 mini_ranger_kms.cc:401] 127.0.98.20:59067/kms/v1/keys
Jun 20 08:56:59 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781945819, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 08:56:59 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945819, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 08:57:00 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945819, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 08:57:02 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:57:02.254247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40513
--ranger_config_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:02.359830 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:02.360076 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:02.360114 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:02.364089 12608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:02.364149 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:02.364167 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:02.364188 12608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:02.364203 12608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:02.369593 12608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40513
--ranger_config_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:02.370584 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:02.371546 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:02.380210 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:02.380265 12608 server_base.cc:1034] running on GCE node
W20260620 08:57:02.380234 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:02.380368 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:02.381203 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:02.382236 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:02.383409 12608 hybrid_clock.cc:648] HybridClock initialized: now 1781945822383389 us; error 32 us; skew 500 ppm
Jun 20 08:57:02 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:02.391279 12608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:02.392277 12608 webserver.cc:460] Webserver started at http://127.0.98.62:44299/ using document root <none> and password file <none>
I20260620 08:57:02.392804 12608 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:02.392861 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:02.393070 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:57:02 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:02.448609 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d"
format_stamp: "Formatted at 2026-06-20 08:57:02 on dist-test-slave-fb1v"
server_key: "84b9d22a70d653f60662e9e3d902dd2f"
server_key_iv: "5dd6b93257b776f0a94213936121fce5"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:02.449195 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d"
format_stamp: "Formatted at 2026-06-20 08:57:02 on dist-test-slave-fb1v"
server_key: "84b9d22a70d653f60662e9e3d902dd2f"
server_key_iv: "5dd6b93257b776f0a94213936121fce5"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:02.452937 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:57:02.469195 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:02.470675 12608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 08:57:02.470809 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d"
format_stamp: "Formatted at 2026-06-20 08:57:02 on dist-test-slave-fb1v"
server_key: "84b9d22a70d653f60662e9e3d902dd2f"
server_key_iv: "5dd6b93257b776f0a94213936121fce5"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:02.470920 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:02.513123 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:02.515897 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:02.516077 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:02.522681 12608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40513
I20260620 08:57:02.522737 12675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40513 every 8 connection(s)
I20260620 08:57:02.523638 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:02.526515 12676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:02.531731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 12608
I20260620 08:57:02.531795 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Bootstrap starting.
I20260620 08:57:02.531857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:57:02.532135 392 external_mini_cluster.cc:1428] Setting key ae93f8005afc79dc2c48c3c9f328f705
I20260620 08:57:02.534325 12676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:02.535013 12676 log.cc:824] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:02.537046 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: No bootstrap required, opened a new log
I20260620 08:57:02.541198 12676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:02.541414 12676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 20 08:57:02 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:02.541452 12676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403d66e4f65a4d9cbc5eeaa2f2273a0d, State: Initialized, Role: FOLLOWER
I20260620 08:57:02.541869 12676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:02.542038 12676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:02.542100 12676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:02.542196 12676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:02.543056 12676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:02.543393 12676 leader_election.cc:302] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403d66e4f65a4d9cbc5eeaa2f2273a0d; no voters:
I20260620 08:57:02.543756 12676 leader_election.cc:288] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:57:02.543841 12681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:02.544003 12681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 LEADER]: Becoming Leader. State: Replica: 403d66e4f65a4d9cbc5eeaa2f2273a0d, State: Running, Role: LEADER
I20260620 08:57:02.544343 12681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:02.545006 12676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:02.545941 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } } }
I20260620 08:57:02.546060 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:02.546398 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403d66e4f65a4d9cbc5eeaa2f2273a0d. Latest consensus state: current_term: 1 leader_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } } }
I20260620 08:57:02.546562 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:02.547331 12676 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:57:02.547837 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:02.533438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.0.98.62:40513)
0620 08:57:02.534107 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:02.534128 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:02.534190 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:57:02.534956 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:02.534974 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:02.535333 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:02.535716 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:02.536498 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:02.537461 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:02.538601 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:02.539057 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:02.539182 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:02.542342 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:02.542349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:02.542365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:02.542401 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:02.544215 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:02.544733 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:02.544739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:02.544745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:02.544849 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:02.545141 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:02.545144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:02.545145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:02.545679 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:02.546577 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:02.546724 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:57:02.546999 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":158,"threads_started":1}
I20260620 08:57:02.547912 12690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:02.553824 12690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:02.559665 12690 catalog_manager.cc:1347] Generated new cluster ID: b568eeaa7f044c4293932cafbe3a47fc
I20260620 08:57:02.559815 12690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:02.574014 12690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:57:02.575246 12690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:02.585207 12690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Generated new TSK 0
I20260620 08:57:02.586002 12690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:57:03.278952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40513
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:03.440907 12721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:03.441267 12721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:03.441330 12721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:03.446043 12721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:03.446169 12721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:03.446290 12721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:03.453393 12721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:03.454454 12721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:03.455423 12721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:03.466806 12728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:03.466943 12729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:03.466820 12731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:03.466991 12721 server_base.cc:1034] running on GCE node
I20260620 08:57:03.467809 12721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:03.468436 12721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:03.469627 12721 hybrid_clock.cc:648] HybridClock initialized: now 1781945823469604 us; error 36 us; skew 500 ppm
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:03.472347 12721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:03.473296 12721 webserver.cc:460] Webserver started at http://127.0.98.1:38181/ using document root <none> and password file <none>
I20260620 08:57:03.473852 12721 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:03.473922 12721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:03.474108 12721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:03.495792 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "cb12fafc81b846c887a1e749ab27fbd9"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "93e8f90c3bc4e593a9ba687f9ac444fd"
server_key_iv: "1e9133f6127f7511bced756c20d67e6c"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:03.496436 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cb12fafc81b846c887a1e749ab27fbd9"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "93e8f90c3bc4e593a9ba687f9ac444fd"
server_key_iv: "1e9133f6127f7511bced756c20d67e6c"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:03.500123 12721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:57:03.519994 12738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:03.521716 12721 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 08:57:03.521855 12721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "cb12fafc81b846c887a1e749ab27fbd9"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "93e8f90c3bc4e593a9ba687f9ac444fd"
server_key_iv: "1e9133f6127f7511bced756c20d67e6c"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:03.521986 12721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:03.547816 12721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:03.551488 12721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:03.551726 12721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:03.552469 12721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:03.553638 12721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:03.553691 12721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:03.553730 12721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:03.553920 12721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:03.564513 12721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43803
I20260620 08:57:03.565798 12721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:57:03.572831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 12721
I20260620 08:57:03.572956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:57:03.573243 392 external_mini_cluster.cc:1428] Setting key b9c2d32611eecfb983904255b0ee6ed7
I20260620 08:57:03.575937 12853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43803 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:57:03.599148 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:03.570642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41993 (local address 127.0.98.62:40513)
0620 08:57:03.583872 (+ 13230us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:03.583882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:03.584158 (+ 276us) server_negotiation.cc:407] Connection header received
0620 08:57:03.585130 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:03.585134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:03.585210 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:03.585292 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:03.586778 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.587641 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.589326 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.589677 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.589752 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:03.594908 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:03.594912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:03.594915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:03.594943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:03.596732 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:03.597334 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:03.597337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:03.597339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:03.597426 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:03.597806 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:03.597809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:03.597811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:03.598025 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:03.598095 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:03.598824 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 08:57:03.598960 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13129,"thread_start_us":155,"threads_started":1}
I20260620 08:57:03.601028 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:03.571575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40513 (local address 127.0.98.1:41993)
0620 08:57:03.584001 (+ 12426us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:03.584056 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:03.584901 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:03.585604 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:03.585615 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:03.585951 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:03.586611 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:03.586630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.587802 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.587806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:03.589189 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:03.589201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.589968 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.589972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:03.590169 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:03.590846 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:03.590868 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:03.594696 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:03.596866 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:03.596872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:03.596889 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:03.597223 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:03.597524 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:03.597526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:03.597528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:03.597694 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:03.598184 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:03.598190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:03.598538 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 08:57:03.599577 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0620 08:57:03.599958 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12135,"thread_start_us":137,"threads_started":1}
I20260620 08:57:03.602706 12854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40513
I20260620 08:57:03.602927 12854 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:03.603392 12854 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:03.605065 12637 ts_manager.cc:194] Registered new tserver with Master: cb12fafc81b846c887a1e749ab27fbd9 (127.0.98.1:43803)
I20260620 08:57:03.606739 12637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:57:03.614315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40513
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 08:57:03.747092 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:03.747334 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:03.747393 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:03.751394 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:03.751449 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:03.751530 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:03.756719 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:03.757846 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:03.758944 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:03.768061 12867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:03.768177 12866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:03.768265 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:03.768479 12860 server_base.cc:1034] running on GCE node
I20260620 08:57:03.768827 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:03.769412 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:03.770710 12860 hybrid_clock.cc:648] HybridClock initialized: now 1781945823770667 us; error 36 us; skew 500 ppm
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:03.773429 12860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:03.774351 12860 webserver.cc:460] Webserver started at http://127.0.98.2:44677/ using document root <none> and password file <none>
I20260620 08:57:03.774919 12860 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:03.774984 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:03.775219 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:03.795295 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa0523a5e24641c7aed6d51563fde150"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "32e4cb60640ab28429e1a2cc5ca8a827"
server_key_iv: "a9d29378ebb75b49919fbe5aefc0cdfc"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:03.795933 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa0523a5e24641c7aed6d51563fde150"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "32e4cb60640ab28429e1a2cc5ca8a827"
server_key_iv: "a9d29378ebb75b49919fbe5aefc0cdfc"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:03.799561 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 08:57:03.818670 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:03.820364 12860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 08:57:03.820488 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "fa0523a5e24641c7aed6d51563fde150"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "32e4cb60640ab28429e1a2cc5ca8a827"
server_key_iv: "a9d29378ebb75b49919fbe5aefc0cdfc"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:03.820623 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:03.858908 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:03.862577 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:03.862805 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:03.863507 12860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:03.865007 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:03.865365 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 08:57:03.865518 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:03.865600 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:03.879114 12860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38033
I20260620 08:57:03.880370 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:03.880965 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38033 every 8 connection(s)
I20260620 08:57:03.886485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 12860
I20260620 08:57:03.886605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:57:03.886885 392 external_mini_cluster.cc:1428] Setting key 18cee14a4e2098ae03cb88e67682820d
Jun 20 08:57:03 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:03.904975 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:03.884743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42797 (local address 127.0.98.62:40513)
0620 08:57:03.884869 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:03.884873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:03.885382 (+ 509us) server_negotiation.cc:407] Connection header received
0620 08:57:03.890539 (+ 5157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:03.890543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:03.890594 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:03.890647 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:03.892593 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.893384 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.896698 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.897062 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.897145 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:03.900703 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:03.900708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:03.900710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:03.900740 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:03.902392 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:03.903039 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:03.903043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:03.903045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:03.903135 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:03.903544 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:03.903548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:03.903549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:03.903834 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:03.903924 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:03.904644 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 08:57:03.904798 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:57:03.906908 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:03.883655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40513 (local address 127.0.98.2:42797)
0620 08:57:03.885230 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:03.885284 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:03.890292 (+ 5008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:03.891084 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:03.891094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:03.891440 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:03.892426 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:03.892445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.893541 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.893544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:03.895008 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:03.895019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:03.897363 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:03.897367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:03.897583 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:03.898737 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:03.898758 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:03.900493 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:03.902534 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:03.902539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:03.902557 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:03.902920 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:03.903244 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:03.903247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:03.903249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:03.903433 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:03.903994 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:03.904001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:03.904354 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 08:57:03.905468 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0620 08:57:03.905838 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":141,"threads_started":1}
I20260620 08:57:03.908558 12990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40513
I20260620 08:57:03.908807 12990 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:03.909395 12990 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:03.911388 12637 ts_manager.cc:194] Registered new tserver with Master: fa0523a5e24641c7aed6d51563fde150 (127.0.98.2:38033)
I20260620 08:57:03.912061 12637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:57:03.936757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40513
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:04.098002 12995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:04.098733 12995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:04.098852 12995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:04.105054 12995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:04.105185 12995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:04.105311 12995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:04.112812 12995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:04.114431 12995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:04.116106 12995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:04.129444 13001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:04.129997 13000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:04.130263 13003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:04.131489 12995 server_base.cc:1034] running on GCE node
I20260620 08:57:04.132221 12995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:04.132951 12995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:04.134111 12995 hybrid_clock.cc:648] HybridClock initialized: now 1781945824134051 us; error 72 us; skew 500 ppm
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:04.137542 12995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:04.138516 12995 webserver.cc:460] Webserver started at http://127.0.98.3:40881/ using document root <none> and password file <none>
I20260620 08:57:04.139153 12995 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:04.139215 12995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:04.139415 12995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:04.164868 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "e97e65e7b16746a1a3fdb4ca51534f9a"
format_stamp: "Formatted at 2026-06-20 08:57:04 on dist-test-slave-fb1v"
server_key: "6227b177253e638534e5e4eec7986cb4"
server_key_iv: "646d3b071b1b80fd8a5753f251f77712"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:04.165721 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e97e65e7b16746a1a3fdb4ca51534f9a"
format_stamp: "Formatted at 2026-06-20 08:57:04 on dist-test-slave-fb1v"
server_key: "6227b177253e638534e5e4eec7986cb4"
server_key_iv: "646d3b071b1b80fd8a5753f251f77712"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:04.179704 12995 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:04.198989 13010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.200730 12995 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 08:57:04.200908 12995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "e97e65e7b16746a1a3fdb4ca51534f9a"
format_stamp: "Formatted at 2026-06-20 08:57:04 on dist-test-slave-fb1v"
server_key: "6227b177253e638534e5e4eec7986cb4"
server_key_iv: "646d3b071b1b80fd8a5753f251f77712"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:04.201033 12995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:04.220269 12995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:04.234735 12995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:04.234967 12995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:04.235739 12995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:04.236850 12995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:04.236905 12995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.236963 12995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:04.236989 12995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.259387 12995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40925
I20260620 08:57:04.261101 12995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:57:04.268234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 12995
I20260620 08:57:04.268345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:57:04.268666 392 external_mini_cluster.cc:1428] Setting key 480d9b5d0f1449af1ecfcec4edb2469e
I20260620 08:57:04.264500 13123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40925 every 8 connection(s)
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:04.300529 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:04.275168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54273 (local address 127.0.98.62:40513)
0620 08:57:04.275301 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:04.275307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:04.283781 (+ 8474us) server_negotiation.cc:407] Connection header received
0620 08:57:04.285219 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:04.285224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:04.285286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:04.285347 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:04.287218 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.288064 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.289817 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.290180 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.290261 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:04.295537 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:04.295542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:04.295545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:04.295578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:04.297524 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:04.298355 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:04.298359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:04.298361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:04.298458 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:04.299042 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:04.299046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:04.299048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:04.299308 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:04.299386 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:04.300198 (+ 812us) server_negotiation.cc:299] Negotiation successful
0620 08:57:04.300341 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:57:04.303042 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:04.273925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40513 (local address 127.0.98.3:54273)
0620 08:57:04.283497 (+ 9572us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:04.283585 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:04.284911 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:04.285864 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:04.285875 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:04.286289 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:04.287025 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:04.287048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.288243 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.288247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:04.289658 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:04.289673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.290491 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.290496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:04.290708 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:04.291478 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:04.291503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:04.295300 (+ 3797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:04.297719 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:04.297726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:04.297746 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:04.298221 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:04.298588 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:04.298592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:04.298594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:04.298852 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:04.299476 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:04.299484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:04.299870 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 08:57:04.301079 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0620 08:57:04.301450 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9247,"thread_start_us":186,"threads_started":1}
I20260620 08:57:04.304677 13124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40513
I20260620 08:57:04.304929 13124 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:04.305575 13124 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:04.308521 12637 ts_manager.cc:194] Registered new tserver with Master: e97e65e7b16746a1a3fdb4ca51534f9a (127.0.98.3:40925)
I20260620 08:57:04.309270 12637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54273
I20260620 08:57:04.319866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:57:04.320598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 12721
I20260620 08:57:04.327473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 12860
I20260620 08:57:04.334475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 12995
I20260620 08:57:04.340327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 12608
I20260620 08:57:04.347854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40513
--webserver_interface=127.0.98.62
--webserver_port=44299
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40513
--ranger_config_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:04.459515 13127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:04.459807 13127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:04.459848 13127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:04.463835 13127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:04.463893 13127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:04.463912 13127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:04.463927 13127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:04.463941 13127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:04.469336 13127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40513
--ranger_config_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:04.470310 13127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:04.471282 13127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:04.479578 13133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:04.479848 13135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:04.479893 13132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:04.480126 13127 server_base.cc:1034] running on GCE node
I20260620 08:57:04.480635 13127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:04.481698 13127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:04.482877 13127 hybrid_clock.cc:648] HybridClock initialized: now 1781945824482852 us; error 36 us; skew 500 ppm
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:04.485569 13127 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:04.486393 13127 webserver.cc:460] Webserver started at http://127.0.98.62:44299/ using document root <none> and password file <none>
I20260620 08:57:04.486893 13127 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:04.486955 13127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:04.532366 13127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:57:04.534561 13143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.535995 13127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:57:04.536120 13127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d"
format_stamp: "Formatted at 2026-06-20 08:57:02 on dist-test-slave-fb1v"
server_key: "84b9d22a70d653f60662e9e3d902dd2f"
server_key_iv: "5dd6b93257b776f0a94213936121fce5"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:04.536665 13127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:04.550518 13127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:04.553342 13127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:04.553501 13127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:04.560273 13127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40513
I20260620 08:57:04.560292 13195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40513 every 8 connection(s)
I20260620 08:57:04.561334 13127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:04.564764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13127
I20260620 08:57:04.565290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43803
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40513
--webserver_port=38181
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:04.565987 13196 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:57:04.567873 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Bootstrap starting.
I20260620 08:57:04.583945 13196 log.cc:824] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:04.590338 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:04.590744 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Bootstrap complete.
I20260620 08:57:04.594588 13196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:04.594962 13196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403d66e4f65a4d9cbc5eeaa2f2273a0d, State: Initialized, Role: FOLLOWER
I20260620 08:57:04.595399 13196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:04.595518 13196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:04.595582 13196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:04.595737 13196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:57:04.597581 13196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:04.597887 13196 leader_election.cc:302] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403d66e4f65a4d9cbc5eeaa2f2273a0d; no voters:
I20260620 08:57:04.598174 13196 leader_election.cc:288] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:57:04.598277 13200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:57:04.598467 13200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [term 2 LEADER]: Becoming Leader. State: Replica: 403d66e4f65a4d9cbc5eeaa2f2273a0d, State: Running, Role: LEADER
I20260620 08:57:04.598821 13200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } }
I20260620 08:57:04.599490 13196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:04.600271 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403d66e4f65a4d9cbc5eeaa2f2273a0d. Latest consensus state: current_term: 2 leader_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } } }
I20260620 08:57:04.600391 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:04.600672 13201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403d66e4f65a4d9cbc5eeaa2f2273a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40513 } } }
I20260620 08:57:04.600754 13201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:04.601649 13196 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:57:04.602065 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:04.608031 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:04.608974 13209 catalog_manager.cc:1259] Loaded cluster ID: b568eeaa7f044c4293932cafbe3a47fc
I20260620 08:57:04.609120 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:04.612320 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:04.613080 13209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Loaded TSK: 0
I20260620 08:57:04.614346 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:57:04.712106 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:04.712437 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:04.712515 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:04.718072 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:04.718140 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:04.718223 13198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:04.723690 13198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38181
--tserver_master_addrs=127.0.98.62:40513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:04.724727 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:04.725768 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:04.735495 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:04.735622 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:04.735502 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:04.736529 13198 server_base.cc:1034] running on GCE node
I20260620 08:57:04.736943 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:04.737614 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:04.741081 13198 hybrid_clock.cc:648] HybridClock initialized: now 1781945824741032 us; error 59 us; skew 500 ppm
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:04.748128 13198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:04.749337 13198 webserver.cc:460] Webserver started at http://127.0.98.1:38181/ using document root <none> and password file <none>
I20260620 08:57:04.750038 13198 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:04.750155 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:04.774873 13198 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:57:04.777915 13244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.779407 13198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:57:04.779534 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "cb12fafc81b846c887a1e749ab27fbd9"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "93e8f90c3bc4e593a9ba687f9ac444fd"
server_key_iv: "1e9133f6127f7511bced756c20d67e6c"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:04.780264 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:04.817164 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:04.821534 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:04.821779 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:04.822533 13198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:04.823587 13198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:04.823676 13198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.823766 13198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:04.823800 13198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:04.837630 13198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43803
I20260620 08:57:04.838122 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43803 every 8 connection(s)
I20260620 08:57:04.839075 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:57:04.846590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13198
I20260620 08:57:04.847060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38033
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40513
--webserver_port=44677
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:57:04 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:04.858101 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:04.841576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40513 (local address 127.0.98.1:46779)
0620 08:57:04.842135 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:04.842186 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:04.842998 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:04.844394 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:04.844402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:04.844779 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:04.845250 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:04.845268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.846656 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.846660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:04.847805 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:04.847817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.848745 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.848749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:04.848917 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:04.849566 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:04.849583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:04.851981 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:04.854434 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:04.854441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:04.854454 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:04.854803 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:04.855216 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:04.855220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:04.855222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:04.855389 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:04.856229 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:04.856235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:04.856536 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 08:57:04.856736 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 08:57:04.857069 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":137,"threads_started":1}
I20260620 08:57:04.859889 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:04.841502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46779 (local address 127.0.98.62:40513)
0620 08:57:04.843377 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:04.843393 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:04.843424 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:57:04.843669 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:04.843682 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:04.843997 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:04.844316 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:04.845544 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.846525 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.847951 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:04.848569 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:04.848722 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:04.852237 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:04.852244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:04.852262 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:04.852296 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:04.854290 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:04.854941 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:04.854947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:04.854952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:04.855094 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:04.855501 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:04.855504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:04.855506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:04.856034 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:04.856164 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:04.858726 (+ 2562us) server_negotiation.cc:299] Negotiation successful
0620 08:57:04.859011 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1742,"thread_start_us":116,"threads_started":1}
I20260620 08:57:04.862412 13358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40513
I20260620 08:57:04.862653 13358 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:04.863221 13358 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
W20260620 08:57:04.865561 13358 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40513 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:57:04.865685 13358 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:04.865861 13358 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:04.867241 13358 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:04.867393 13358 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
W20260620 08:57:04.868134 13358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:57:04.983592 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:04.983942 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:04.984021 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:04.989352 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:04.989434 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:04.989526 13362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:04.996953 13362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44677
--tserver_master_addrs=127.0.98.62:40513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:04.998294 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:04.999598 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:05.012115 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:05.012269 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:05.012137 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:05.013396 13362 server_base.cc:1034] running on GCE node
I20260620 08:57:05.013821 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:05.014530 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:05.017848 13362 hybrid_clock.cc:648] HybridClock initialized: now 1781945825017817 us; error 104 us; skew 500 ppm
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:05.021778 13362 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:05.022919 13362 webserver.cc:460] Webserver started at http://127.0.98.2:44677/ using document root <none> and password file <none>
I20260620 08:57:05.023659 13362 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:05.023783 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:05.052934 13362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:57:05.057439 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:05.058984 13362 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 08:57:05.059126 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "fa0523a5e24641c7aed6d51563fde150"
format_stamp: "Formatted at 2026-06-20 08:57:03 on dist-test-slave-fb1v"
server_key: "32e4cb60640ab28429e1a2cc5ca8a827"
server_key_iv: "a9d29378ebb75b49919fbe5aefc0cdfc"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:05.059900 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:05.076741 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:05.080096 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:05.080295 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:05.081193 13362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:05.082234 13362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:05.082298 13362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:05.082362 13362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:05.082393 13362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:05.096557 13362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38033
I20260620 08:57:05.096747 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38033 every 8 connection(s)
I20260620 08:57:05.097765 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:05.098016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13362
I20260620 08:57:05.098459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40925
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40513
--webserver_port=40881
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:05.120633 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:05.102620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44095 (local address 127.0.98.62:40513)
0620 08:57:05.102755 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:05.102760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:05.103359 (+ 599us) server_negotiation.cc:407] Connection header received
0620 08:57:05.104612 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:05.104616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:05.104676 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:05.104791 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:05.106225 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.107071 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.108907 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.109206 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.109350 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:05.116320 (+ 6970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:05.116325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:05.116328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:05.116370 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:05.118131 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:05.118768 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:05.118771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:05.118773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:05.118860 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:05.119253 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:05.119256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:05.119258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:05.119487 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:05.119574 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:05.120285 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 08:57:05.120430 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:57:05.122498 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:05.101575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40513 (local address 127.0.98.2:44095)
0620 08:57:05.103169 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:05.103228 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:05.104351 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:05.105007 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:05.105018 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:05.105369 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:05.106050 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:05.106072 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.107216 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.107220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:05.108654 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:05.108665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.109414 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.109418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:05.109626 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:05.110488 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:05.110519 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:05.116039 (+ 5520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:05.118269 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:05.118276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:05.118295 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:05.118658 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:05.118964 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:05.118967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:05.118970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:05.119150 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:05.119664 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:05.119671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:05.120011 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 08:57:05.121093 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0620 08:57:05.121450 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":135,"threads_started":1}
I20260620 08:57:05.124238 13492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40513
I20260620 08:57:05.124470 13492 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.125018 13492 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
W20260620 08:57:05.126679 13492 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40513 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:57:05.126801 13492 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.126968 13492 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:05.127801 13492 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.127969 13492 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
W20260620 08:57:05.128698 13492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:57:05.234966 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:05.235248 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:05.235312 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:05.240936 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:05.241017 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:05.241101 13494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:05.248696 13494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40881
--tserver_master_addrs=127.0.98.62:40513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:05.249869 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:05.250967 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:05.262897 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:05.263330 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:05.263937 13494 server_base.cc:1034] running on GCE node
W20260620 08:57:05.264706 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:05.265278 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:05.266433 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:05.267688 13494 hybrid_clock.cc:648] HybridClock initialized: now 1781945825267647 us; error 42 us; skew 500 ppm
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:05.270430 13494 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:05.271378 13494 webserver.cc:460] Webserver started at http://127.0.98.3:40881/ using document root <none> and password file <none>
I20260620 08:57:05.271997 13494 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:05.272081 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:05.290340 13494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:57:05.293745 13512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:05.296336 13494 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 08:57:05.296485 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "e97e65e7b16746a1a3fdb4ca51534f9a"
format_stamp: "Formatted at 2026-06-20 08:57:04 on dist-test-slave-fb1v"
server_key: "6227b177253e638534e5e4eec7986cb4"
server_key_iv: "646d3b071b1b80fd8a5753f251f77712"
server_key_version: "kuduclusterkey@0"
I20260620 08:57:05.297235 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:05.318954 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:05.323055 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:05.323275 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:05.324015 13494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:05.324985 13494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:05.325035 13494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:05.325086 13494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:05.325114 13494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:05.335434 13494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40925
I20260620 08:57:05.335701 13625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40925 every 8 connection(s)
I20260620 08:57:05.337114 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:05.353845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13494
I20260620 08:57:05.354530 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:05.339781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47669 (local address 127.0.98.62:40513)
0620 08:57:05.339907 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:05.339912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:05.340860 (+ 948us) server_negotiation.cc:407] Connection header received
0620 08:57:05.341907 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:05.341911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:05.341962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:05.342004 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:05.343196 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.343830 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.345490 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.345792 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.345891 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:05.348876 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:05.348880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:05.348883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:05.348913 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:05.350713 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:05.352053 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:05.352057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:05.352059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:05.352165 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:05.352663 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:05.352666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:05.352668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:05.352912 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:05.352999 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:05.354154 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0620 08:57:05.354314 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:57:05.357833 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:05.340210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40513 (local address 127.0.98.3:47669)
0620 08:57:05.340670 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:05.340720 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:05.341691 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:05.342315 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:05.342322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:05.342577 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:05.343053 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:05.343065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.343983 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.343986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:05.345341 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:05.345352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.345956 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.345961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:05.346171 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:05.346854 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:05.346876 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:05.348662 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:05.351528 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:05.351535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:05.351554 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:05.351930 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:05.352286 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:05.352289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:05.352291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:05.352481 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:05.353110 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:05.353117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:05.353680 (+ 563us) client_negotiation.cc:769] Sending connection context
0620 08:57:05.356312 (+ 2632us) client_negotiation.cc:240] Negotiation successful
0620 08:57:05.356696 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260620 08:57:05.359674 13626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40513
I20260620 08:57:05.359948 13626 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.360664 13626 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
W20260620 08:57:05.363020 13626 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40513 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:57:05.363301 13626 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.363579 13626 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:05.364729 13626 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.364917 13626 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
W20260620 08:57:05.365856 13626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:57:05.371007 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:05.355878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.98.62:40513)
0620 08:57:05.356251 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:05.356256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:05.359325 (+ 3069us) server_negotiation.cc:407] Connection header received
0620 08:57:05.359383 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:05.359387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:05.359443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:05.359535 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:05.361552 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.362361 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.364058 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:05.364412 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:05.364520 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:05.367691 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:05.367695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:05.367697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:05.367724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:05.369274 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:05.369728 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:05.369732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:05.369734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:05.369827 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:05.370126 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:05.370129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:05.370131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:05.370368 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:05.370585 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:05.370727 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 08:57:05.370845 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
Jun 20 08:57:05 dist-test-slave-fb1v krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:05.868381 13358 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:05.868698 13358 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:05.869881 13160 ts_manager.cc:194] Registered new tserver with Master: cb12fafc81b846c887a1e749ab27fbd9 (127.0.98.1:43803)
I20260620 08:57:05.871543 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46779
Jun 20 08:57:06 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:06.128942 13492 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:06.129249 13492 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:06.132226 13160 ts_manager.cc:194] Registered new tserver with Master: fa0523a5e24641c7aed6d51563fde150 (127.0.98.2:38033)
I20260620 08:57:06.133004 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44095
Jun 20 08:57:06 dist-test-slave-fb1v krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 08:57:06.366091 13626 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:06.366387 13626 heartbeater.cc:511] Master 127.0.98.62:40513 requested a full tablet report, sending...
I20260620 08:57:06.367056 13160 ts_manager.cc:194] Registered new tserver with Master: e97e65e7b16746a1a3fdb4ca51534f9a (127.0.98.3:40925)
I20260620 08:57:06.367839 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47669
I20260620 08:57:06.376605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:57:06.387851 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.378668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.62:40513)
0620 08:57:06.378922 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.378926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.379051 (+ 125us) server_negotiation.cc:407] Connection header received
0620 08:57:06.379174 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.379177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.379227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.379321 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:06.380128 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.380983 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.381916 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.382161 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.382247 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.383797 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:06.383801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:06.383804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:06.383831 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:06.385714 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:06.386119 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:06.386123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:06.386124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:06.386221 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:06.386578 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:06.386582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:06.386584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:06.386795 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:06.386918 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.387520 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.387670 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
I20260620 08:57:06.392027 13160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:57:06.397006 13160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:57:06.432061 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.421361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40925 (local address 127.0.0.1:47920)
0620 08:57:06.424868 (+ 3507us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.424925 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:06.425053 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:06.426287 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:06.426293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:06.426316 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:06.426449 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.426456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.427823 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.427826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.429247 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.429257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.430473 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.430477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.431598 (+ 1121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.431670 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 08:57:06.431767 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 08:57:06.431895 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3329,"mutex_wait_us":1000,"thread_start_us":89,"threads_started":1}
I20260620 08:57:06.432693 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.422173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.0.98.3:40925)
0620 08:57:06.425597 (+ 3424us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.425604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.425628 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:57:06.425701 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.425706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.425926 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.426089 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:06.426584 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.427709 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.429374 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.430186 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.430282 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.430419 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.432443 (+ 2024us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.432550 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3321,"thread_start_us":115,"threads_started":1}
I20260620 08:57:06.435946 13543 tablet_service.cc:1467] Processing CreateTablet for tablet daec26a51feb448e97c9cbe6755c429c (DEFAULT_TABLE table=test-table [id=161e5e8ecff942a28438c11eb3234fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:06.436985 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.423593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38033 (local address 127.0.0.1:37046)
0620 08:57:06.427306 (+ 3713us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.427319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:06.427409 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:06.428125 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:06.428128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:06.428144 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:06.428232 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.428237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.429525 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.429529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.430614 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.430623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.436598 (+ 5975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.436601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.436685 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.436711 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 08:57:06.436773 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 08:57:06.436844 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3650,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 08:57:06.437119 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daec26a51feb448e97c9cbe6755c429c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:06.437564 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.424015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.0.98.2:38033)
0620 08:57:06.424610 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.424618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.427494 (+ 2876us) server_negotiation.cc:407] Connection header received
0620 08:57:06.427583 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.427588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.427832 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.427979 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:06.428354 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.429417 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.430740 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.431668 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.431886 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.432016 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.437299 (+ 5283us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.437416 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":110,"threads_started":1}
I20260620 08:57:06.437987 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.422957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43803 (local address 127.0.0.1:57128)
0620 08:57:06.425291 (+ 2334us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.425302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:06.425375 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:06.431348 (+ 5973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:06.431351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:06.431369 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:06.431450 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.431455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.435308 (+ 3853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.435312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.436541 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.436553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.437638 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.437642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.437729 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.437752 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:57:06.437798 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:57:06.437865 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"thread_start_us":83,"threads_started":1}
I20260620 08:57:06.438635 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.423110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.0.98.1:43803)
0620 08:57:06.430728 (+ 7618us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.430736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.430759 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:57:06.430832 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.430837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.431037 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.431186 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:06.433601 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.434745 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.436709 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.437494 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.438236 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.438373 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.438417 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.438500 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7502,"thread_start_us":61,"threads_started":1}
I20260620 08:57:06.440760 13417 tablet_service.cc:1467] Processing CreateTablet for tablet daec26a51feb448e97c9cbe6755c429c (DEFAULT_TABLE table=test-table [id=161e5e8ecff942a28438c11eb3234fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:06.441833 13417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daec26a51feb448e97c9cbe6755c429c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:06.445732 13291 tablet_service.cc:1467] Processing CreateTablet for tablet daec26a51feb448e97c9cbe6755c429c (DEFAULT_TABLE table=test-table [id=161e5e8ecff942a28438c11eb3234fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:06.446815 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daec26a51feb448e97c9cbe6755c429c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:06.457283 13660 tablet_bootstrap.cc:492] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Bootstrap starting.
I20260620 08:57:06.460120 13661 tablet_bootstrap.cc:492] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: Bootstrap starting.
I20260620 08:57:06.461148 13660 tablet_bootstrap.cc:654] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:06.461407 13662 tablet_bootstrap.cc:492] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: Bootstrap starting.
I20260620 08:57:06.462162 13660 log.cc:824] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:06.462371 13661 tablet_bootstrap.cc:654] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:06.463236 13661 log.cc:824] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:06.464066 13662 tablet_bootstrap.cc:654] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:06.464581 13660 tablet_bootstrap.cc:492] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: No bootstrap required, opened a new log
I20260620 08:57:06.464771 13660 ts_tablet_manager.cc:1397] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 08:57:06.464852 13662 log.cc:824] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:06.465423 13661 tablet_bootstrap.cc:492] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: No bootstrap required, opened a new log
I20260620 08:57:06.465600 13661 ts_tablet_manager.cc:1397] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:57:06.467044 13662 tablet_bootstrap.cc:492] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: No bootstrap required, opened a new log
I20260620 08:57:06.467226 13662 ts_tablet_manager.cc:1397] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:57:06.469621 13661 raft_consensus.cc:348] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.470064 13661 raft_consensus.cc:374] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:06.470124 13661 raft_consensus.cc:729] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0523a5e24641c7aed6d51563fde150, State: Initialized, Role: FOLLOWER
I20260620 08:57:06.470039 13660 raft_consensus.cc:348] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.470279 13660 raft_consensus.cc:374] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:06.470348 13660 raft_consensus.cc:729] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97e65e7b16746a1a3fdb4ca51534f9a, State: Initialized, Role: FOLLOWER
I20260620 08:57:06.470644 13661 consensus_queue.cc:260] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.470773 13660 consensus_queue.cc:260] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.471752 13661 ts_tablet_manager.cc:1428] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 08:57:06.471774 13660 ts_tablet_manager.cc:1428] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 08:57:06.472113 13492 heartbeater.cc:503] Master 127.0.98.62:40513 was elected leader, sending a full tablet report...
I20260620 08:57:06.472113 13626 heartbeater.cc:503] Master 127.0.98.62:40513 was elected leader, sending a full tablet report...
I20260620 08:57:06.471962 13662 raft_consensus.cc:348] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.472313 13662 raft_consensus.cc:374] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:06.472398 13662 raft_consensus.cc:729] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb12fafc81b846c887a1e749ab27fbd9, State: Initialized, Role: FOLLOWER
I20260620 08:57:06.472910 13662 consensus_queue.cc:260] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.474452 13662 ts_tablet_manager.cc:1428] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 08:57:06.474808 13358 heartbeater.cc:503] Master 127.0.98.62:40513 was elected leader, sending a full tablet report...
W20260620 08:57:06.589179 13627 tablet.cc:2369] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:06.590147 13668 raft_consensus.cc:482] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:06.590368 13668 raft_consensus.cc:504] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.591841 13668 leader_election.cc:288] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e97e65e7b16746a1a3fdb4ca51534f9a (127.0.98.3:40925), fa0523a5e24641c7aed6d51563fde150 (127.0.98.2:38033)
W20260620 08:57:06.592845 13359 tablet.cc:2369] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:06.599825 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.592988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60923 (local address 127.0.98.3:40925)
0620 08:57:06.593265 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.593270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.593474 (+ 204us) server_negotiation.cc:407] Connection header received
0620 08:57:06.593631 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.593635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.593688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.593773 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:06.594619 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.595408 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.598283 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.598934 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.599013 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.599081 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.599581 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.599687 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260620 08:57:06.600276 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.592240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40925 (local address 127.0.98.1:60923)
0620 08:57:06.593387 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.593403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:06.593531 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:06.594321 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:06.594324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:06.594342 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:06.594483 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.594489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.595560 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.595563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.596904 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.596915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.599218 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.599221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.599330 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.599354 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:57:06.600030 (+ 676us) client_negotiation.cc:240] Negotiation successful
0620 08:57:06.600116 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":108,"threads_started":1}
W20260620 08:57:06.600502 13493 tablet.cc:2369] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:06.601511 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec26a51feb448e97c9cbe6755c429c" candidate_uuid: "cb12fafc81b846c887a1e749ab27fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" is_pre_election: true
I20260620 08:57:06.601859 13580 raft_consensus.cc:2454] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb12fafc81b846c887a1e749ab27fbd9 in term 0.
I20260620 08:57:06.602533 13246 leader_election.cc:302] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb12fafc81b846c887a1e749ab27fbd9, e97e65e7b16746a1a3fdb4ca51534f9a; no voters:
I20260620 08:57:06.602942 13668 raft_consensus.cc:2781] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:57:06.603036 13668 raft_consensus.cc:482] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:57:06.603081 13668 raft_consensus.cc:3037] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:06.604517 13668 raft_consensus.cc:504] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.605067 13668 leader_election.cc:288] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [CANDIDATE]: Term 1 election: Requested vote from peers e97e65e7b16746a1a3fdb4ca51534f9a (127.0.98.3:40925), fa0523a5e24641c7aed6d51563fde150 (127.0.98.2:38033)
I20260620 08:57:06.605876 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec26a51feb448e97c9cbe6755c429c" candidate_uuid: "cb12fafc81b846c887a1e749ab27fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a"
I20260620 08:57:06.606221 13580 raft_consensus.cc:3037] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:06.607508 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.593133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55957 (local address 127.0.98.2:38033)
0620 08:57:06.593350 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.593354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.593866 (+ 512us) server_negotiation.cc:407] Connection header received
0620 08:57:06.593957 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.593960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.594150 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.594227 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:06.599797 (+ 5570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.600487 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.606089 (+ 5602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.606726 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.606817 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.606894 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.607278 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.607373 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260620 08:57:06.607518 13580 raft_consensus.cc:2454] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb12fafc81b846c887a1e749ab27fbd9 in term 1.
I20260620 08:57:06.611850 13246 leader_election.cc:302] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb12fafc81b846c887a1e749ab27fbd9, e97e65e7b16746a1a3fdb4ca51534f9a; no voters:
I20260620 08:57:06.612130 13668 raft_consensus.cc:2781] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:06.612368 13668 raft_consensus.cc:686] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [term 1 LEADER]: Becoming Leader. State: Replica: cb12fafc81b846c887a1e749ab27fbd9, State: Running, Role: LEADER
I20260620 08:57:06.612779 13668 consensus_queue.cc:237] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } }
I20260620 08:57:06.619915 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.592618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38033 (local address 127.0.98.1:55957)
0620 08:57:06.593778 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.593790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:06.593877 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:06.599448 (+ 5571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:06.599452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:06.599478 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:06.599600 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.599624 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.601286 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.601289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.605909 (+ 4620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:06.605923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.607000 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.607004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:06.607106 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.607131 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 08:57:06.619583 (+ 12452us) client_negotiation.cc:240] Negotiation successful
0620 08:57:06.619740 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":86,"threads_started":1}
I20260620 08:57:06.620792 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec26a51feb448e97c9cbe6755c429c" candidate_uuid: "cb12fafc81b846c887a1e749ab27fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0523a5e24641c7aed6d51563fde150" is_pre_election: true
I20260620 08:57:06.621141 13446 raft_consensus.cc:2454] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb12fafc81b846c887a1e749ab27fbd9 in term 0.
I20260620 08:57:06.620620 13158 catalog_manager.cc:5495] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb12fafc81b846c887a1e749ab27fbd9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cb12fafc81b846c887a1e749ab27fbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb12fafc81b846c887a1e749ab27fbd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 } health_report { overall_health: UNKNOWN } } }
I20260620 08:57:06.621495 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec26a51feb448e97c9cbe6755c429c" candidate_uuid: "cb12fafc81b846c887a1e749ab27fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0523a5e24641c7aed6d51563fde150"
I20260620 08:57:06.621603 13445 raft_consensus.cc:3037] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:06.623003 13445 raft_consensus.cc:2454] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb12fafc81b846c887a1e749ab27fbd9 in term 1.
I20260620 08:57:06.667037 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.660412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.98.1:43803)
0620 08:57:06.660556 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:06.660561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:06.660593 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:57:06.660734 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:06.660738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:06.660810 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:06.660914 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:06.661305 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.662067 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.663978 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:06.664337 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:06.664456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:06.664932 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:06.665520 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:57:06.665646 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:06.666618 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 08:57:06.666740 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:57:06.674423 13445 raft_consensus.cc:1261] T daec26a51feb448e97c9cbe6755c429c P fa0523a5e24641c7aed6d51563fde150 [term 1 FOLLOWER]: Refusing update from remote peer cb12fafc81b846c887a1e749ab27fbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:06.675262 13671 consensus_queue.cc:1035] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa0523a5e24641c7aed6d51563fde150" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:06.682744 13580 raft_consensus.cc:1261] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 1 FOLLOWER]: Refusing update from remote peer cb12fafc81b846c887a1e749ab27fbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:06.683518 13671 consensus_queue.cc:1035] T daec26a51feb448e97c9cbe6755c429c P cb12fafc81b846c887a1e749ab27fbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e97e65e7b16746a1a3fdb4ca51534f9a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:06.695220 13675 mvcc.cc:205] Tried to move back new op lower bound from 7298850106050310144 to 7298850105809326080. Current Snapshot: MvccSnapshot[applied={T|T < 7298850106050310144}]
I20260620 08:57:06.702042 13674 mvcc.cc:205] Tried to move back new op lower bound from 7298850106050310144 to 7298850105809326080. Current Snapshot: MvccSnapshot[applied={T|T < 7298850106050310144}]
I20260620 08:57:06.711926 13158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:06.712147 13158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:06.714898 13158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 403d66e4f65a4d9cbc5eeaa2f2273a0d: Sending DeleteTablet for 3 replicas of tablet daec26a51feb448e97c9cbe6755c429c
I20260620 08:57:06.716068 13417 tablet_service.cc:1514] Processing DeleteTablet for tablet daec26a51feb448e97c9cbe6755c429c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37046
I20260620 08:57:06.716063 13543 tablet_service.cc:1514] Processing DeleteTablet for tablet daec26a51feb448e97c9cbe6755c429c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47920
I20260620 08:57:06.716763 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet daec26a51feb448e97c9cbe6755c429c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57128
I20260620 08:57:06.717564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13198
I20260620 08:57:06.718663 13686 tablet_replica.cc:330] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: stopping tablet replica
I20260620 08:57:06.719033 13686 raft_consensus.cc:2229] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:06.719317 13686 raft_consensus.cc:2258] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:06.725522 13686 ts_tablet_manager.cc:1905] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:57:06.726459 13145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43803 (error 108)
I20260620 08:57:06.727289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13362
I20260620 08:57:06.728559 13686 ts_tablet_manager.cc:1918] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:06.728657 13686 log.cc:1196] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/daec26a51feb448e97c9cbe6755c429c
I20260620 08:57:06.728968 13686 ts_tablet_manager.cc:1939] T daec26a51feb448e97c9cbe6755c429c P e97e65e7b16746a1a3fdb4ca51534f9a: Deleting consensus metadata
I20260620 08:57:06.736045 13145 catalog_manager.cc:4841] TS e97e65e7b16746a1a3fdb4ca51534f9a (127.0.98.3:40925): tablet daec26a51feb448e97c9cbe6755c429c (table test-table [id=161e5e8ecff942a28438c11eb3234fec]) successfully deleted
I20260620 08:57:06.736771 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:06.736432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43803 (local address 127.0.0.1:57142)
0620 08:57:06.736611 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.736679 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 08:57:06.736953 13145 catalog_manager.cc:4584] TS cb12fafc81b846c887a1e749ab27fbd9 (127.0.98.1:43803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daec26a51feb448e97c9cbe6755c429c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43803: connect: Connection refused (error 111)
W20260620 08:57:06.738139 13658 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:57:06.737621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38033 (local address 127.0.0.1:37048)
0620 08:57:06.737762 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:06.737776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:06.737864 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:06.738049 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 08:57:06.738692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13494
I20260620 08:57:06.745666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13127
I20260620 08:57:06.752662 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 08:57:07.098152 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 08:57:07.098259 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 08:57:07.437844 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 08:57:07.438 UTC [12194] LOG: received smart shutdown request
2026-06-20 08:57:07.441 UTC [12194] LOG: background worker "logical replication launcher" (PID 12203) exited with exit code 1
2026-06-20 08:57:07.442 UTC [12198] LOG: shutting down
2026-06-20 08:57:07.587 UTC [12194] LOG: database system is shut down
2026-06-20T08:57:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48613 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:57:07 dist-test-slave-fb1v krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:57:07 dist-test-slave-fb1v krb5kdc[13703](info): set up 2 sockets
Jun 20 08:57:07 dist-test-slave-fb1v krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260620 08:57:08.643414 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:57:08 dist-test-slave-fb1v krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:57:08.762318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33747,127.0.98.61:37755,127.0.98.60:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:08.868153 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:08.868396 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:08.868451 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:08.872407 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:08.872460 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:08.872486 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:08.872522 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:08.872553 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:08.877628 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33747,127.0.98.61:37755,127.0.98.60:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:08.878592 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:08.879529 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:57:08.887724 13716 server_base.cc:1034] running on GCE node
W20260620 08:57:08.887676 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:08.887676 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:08.887710 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:08.888420 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:08.889427 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:08.890602 13716 hybrid_clock.cc:648] HybridClock initialized: now 1781945828890580 us; error 32 us; skew 500 ppm
Jun 20 08:57:08 dist-test-slave-fb1v krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:08.893217 13716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:08.894032 13716 webserver.cc:460] Webserver started at http://127.0.98.62:35561/ using document root <none> and password file <none>
I20260620 08:57:08.894524 13716 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:08.894598 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:08.894793 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:08.896777 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8"
format_stamp: "Formatted at 2026-06-20 08:57:08 on dist-test-slave-fb1v"
server_key: "19b285f5af2aa1409efcb212be32e5ab"
server_key_iv: "144f2bc22f560302e0ff84dc57d69cea"
server_key_version: "encryptionkey@0"
I20260620 08:57:08.897297 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8"
format_stamp: "Formatted at 2026-06-20 08:57:08 on dist-test-slave-fb1v"
server_key: "19b285f5af2aa1409efcb212be32e5ab"
server_key_iv: "144f2bc22f560302e0ff84dc57d69cea"
server_key_version: "encryptionkey@0"
I20260620 08:57:08.901298 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 08:57:08.903834 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:08.905076 13716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:08.905210 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8"
format_stamp: "Formatted at 2026-06-20 08:57:08 on dist-test-slave-fb1v"
server_key: "19b285f5af2aa1409efcb212be32e5ab"
server_key_iv: "144f2bc22f560302e0ff84dc57d69cea"
server_key_version: "encryptionkey@0"
I20260620 08:57:08.905324 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:08.921845 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:08.925794 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:08.925956 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:08.933166 13716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33747
I20260620 08:57:08.933249 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33747 every 8 connection(s)
I20260620 08:57:08.934084 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:08.936789 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:08.938076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13716
I20260620 08:57:08.938207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:57:08.938510 392 external_mini_cluster.cc:1428] Setting key 3398afdf85008b6ab4d698389418cf81
I20260620 08:57:08.941143 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 08:57:08 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:08.956544 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:08.943488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.0.98.62:33747)
0620 08:57:08.944013 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:08.944032 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:08.944317 (+ 285us) server_negotiation.cc:407] Connection header received
0620 08:57:08.945508 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:08.945514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:08.945761 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:08.945960 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:08.946974 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:08.948066 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:08.949134 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:08.949498 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:08.949623 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:08.952023 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:08.952029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:08.952041 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:08.952065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:08.953620 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:08.954046 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:08.954050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:08.954051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:08.954147 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:08.954436 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:08.954439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:08.954441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:08.954805 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:08.954920 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:08.955252 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 08:57:08.955521 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":360,"thread_start_us":159,"threads_started":1}
I20260620 08:57:08.956542 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:08.943488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33747 (local address 127.0.0.1:42170)
0620 08:57:08.944146 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:08.944193 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:08.945017 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:08.945979 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:08.945983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:08.946357 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:08.946620 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:08.946634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:08.948185 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:08.948188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:08.949013 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:08.949024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:08.949667 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:08.949671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:08.949785 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:08.950361 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:08.950377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:08.951879 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:08.953723 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:08.953731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:08.953741 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:08.953949 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:08.954249 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:08.954252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:08.954254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:08.954355 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:08.954908 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:08.954914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:08.955040 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 08:57:08.955184 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 08:57:08.955578 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":101,"spinlock_wait_cycles":54528,"thread_start_us":162,"threads_started":1}
I20260620 08:57:08.958765 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:57:08.959452 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:08.959205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59500)
0620 08:57:08.959328 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:08.959392 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 08:57:08.959676 13735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
I20260620 08:57:08.960500 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:08.960310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59510)
0620 08:57:08.960400 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:08.960446 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:57:08.960632 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 08:57:08.962144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37755
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33747,127.0.98.61:37755,127.0.98.60:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:09.009826 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } attempt: 1
I20260620 08:57:09.010728 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.010384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59520)
0620 08:57:09.010555 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.010625 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 08:57:09.011833 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.011513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59528)
0620 08:57:09.011705 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.011763 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 08:57:09.012023 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
W20260620 08:57:09.072373 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:09.072630 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:09.072669 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:09.076648 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:09.076710 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:09.076728 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:09.076743 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:09.076757 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:09.080200 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } attempt: 2
I20260620 08:57:09.081007 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.080684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59530)
0620 08:57:09.080852 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.080920 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 08:57:09.081687 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.081480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59538)
0620 08:57:09.081574 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.081619 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 08:57:09.081835 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37755: connect: Connection refused (error 111)
I20260620 08:57:09.082381 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33747,127.0.98.61:37755,127.0.98.60:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:09.083410 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:09.084425 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:09.093189 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:09.093232 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:09.093202 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:09.093536 13791 server_base.cc:1034] running on GCE node
I20260620 08:57:09.094056 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:09.095158 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:09.096325 13791 hybrid_clock.cc:648] HybridClock initialized: now 1781945829096307 us; error 28 us; skew 500 ppm
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:09.098948 13791 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 08:57:09.099866 13791 webserver.cc:460] Webserver started at http://127.0.98.61:45577/ using document root <none> and password file <none>
I20260620 08:57:09.100384 13791 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:09.100448 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:09.100626 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:09.102658 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data/instance:
uuid: "45702cd90008459695d62f4b002a7fd6"
format_stamp: "Formatted at 2026-06-20 08:57:09 on dist-test-slave-fb1v"
server_key: "e504099a875181207d03f0be2df4d433"
server_key_iv: "05ea03d09a4a59d98d7258d81353cd24"
server_key_version: "encryptionkey@0"
I20260620 08:57:09.103204 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal/instance:
uuid: "45702cd90008459695d62f4b002a7fd6"
format_stamp: "Formatted at 2026-06-20 08:57:09 on dist-test-slave-fb1v"
server_key: "e504099a875181207d03f0be2df4d433"
server_key_iv: "05ea03d09a4a59d98d7258d81353cd24"
server_key_version: "encryptionkey@0"
I20260620 08:57:09.107407 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 08:57:09.109823 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:09.111020 13791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:09.111133 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal
uuid: "45702cd90008459695d62f4b002a7fd6"
format_stamp: "Formatted at 2026-06-20 08:57:09 on dist-test-slave-fb1v"
server_key: "e504099a875181207d03f0be2df4d433"
server_key_iv: "05ea03d09a4a59d98d7258d81353cd24"
server_key_version: "encryptionkey@0"
I20260620 08:57:09.111232 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:09.129608 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:09.132473 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:09.132642 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:09.139385 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37755
I20260620 08:57:09.139405 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37755 every 8 connection(s)
I20260620 08:57:09.140347 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
I20260620 08:57:09.142975 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:09.146567 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:57:09.148447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13791
I20260620 08:57:09.148566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-1/wal/instance
I20260620 08:57:09.148859 392 external_mini_cluster.cc:1428] Setting key cf2e23b0ad7bab0a5729da9407defe19
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 08:57:09.161454 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.148381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.0.98.62:33747)
0620 08:57:09.148498 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.148503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.149486 (+ 983us) server_negotiation.cc:407] Connection header received
0620 08:57:09.150616 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.150618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.150669 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.150752 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.152165 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.152743 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.154358 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.154610 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.154701 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.157369 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.157372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.157374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.157400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.159332 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.159912 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.159915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.159917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.160007 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.160316 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.160320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.160321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.160508 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.160595 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.161162 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.161292 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:57:09.162380 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.148823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33747 (local address 127.0.0.1:42172)
0620 08:57:09.149333 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.149374 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.150384 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.150914 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.150922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.151380 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.152033 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.152050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.152869 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.152873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.154218 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.154230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.154766 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.154771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.154969 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.155585 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.155633 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.157178 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.159476 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.159482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.159495 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.159810 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.160083 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.160086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.160087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.160214 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.160607 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.160614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.160877 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.161039 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.161362 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260620 08:57:09.163815 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:57:09.173024 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.164317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59546)
0620 08:57:09.164467 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.164478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.164546 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.165052 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.165055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.165222 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.165298 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.165303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.166215 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.166218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.166743 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.166750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.167266 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.167269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.167423 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.167864 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.167876 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.169246 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.171312 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.171315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.171317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.171557 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.171902 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.171905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.171907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.171986 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.172534 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.172537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.172619 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.172712 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.172809 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":43}
I20260620 08:57:09.173007 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } attempt: 3
I20260620 08:57:09.173094 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.164426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.98.61:37755)
0620 08:57:09.164744 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.164752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.164768 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:57:09.164800 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.164805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.164943 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.165058 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.165391 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.166118 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.166840 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.167169 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.167260 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.169366 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.169374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.169384 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.169405 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.171189 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.171729 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.171734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.171735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.171808 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.172086 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.172089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.172090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.172380 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.172516 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.172788 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.172907 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 08:57:09.175383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34571
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33747,127.0.98.61:37755,127.0.98.60:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:09.175637 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:57:09.176504 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.176031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58798)
0620 08:57:09.176133 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.176221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 08:57:09.176676 13807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
I20260620 08:57:09.177505 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.177305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58806)
0620 08:57:09.177398 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.177446 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:57:09.177597 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34571: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:57:09.181754 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.173465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.0.98.61:37755)
0620 08:57:09.173560 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.173564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.173669 (+ 105us) server_negotiation.cc:407] Connection header received
0620 08:57:09.173759 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.173761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.173792 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.173858 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.174250 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.174882 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.175867 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.176175 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.176272 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.178486 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.178490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.178491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.178512 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.180306 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.180745 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.180747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.180748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.180811 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.181063 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.181065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.181066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.181225 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.181293 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.181554 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.181656 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:57:09.181828 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.173469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59558)
0620 08:57:09.173618 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.173629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.173707 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.173895 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.173898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.174045 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.174138 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.174143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.175007 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.175010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.175768 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.175780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.176279 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.176282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.176382 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.176815 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.176830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.178368 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.180409 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.180413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.180415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.180659 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.180886 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.180889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.180891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.180984 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.181305 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.181308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.181414 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.181524 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.181667 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 08:57:09.182508 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:57:09.183118 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.182932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58816)
0620 08:57:09.183021 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.183062 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 08:57:09.183722 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.183492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58824)
0620 08:57:09.183602 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.183672 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 08:57:09.183847 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34571: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
I20260620 08:57:09.215010 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } attempt: 1
I20260620 08:57:09.215938 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.215630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58836)
0620 08:57:09.215794 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.215855 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 08:57:09.216585 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.216396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58846)
0620 08:57:09.216490 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.216533 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:57:09.216697 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34571: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
I20260620 08:57:09.226840 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } attempt: 1
I20260620 08:57:09.227726 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.227369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58860)
0620 08:57:09.227534 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.227595 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 08:57:09.228364 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.228174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58874)
0620 08:57:09.228270 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.228314 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:57:09.228482 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34571: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
W20260620 08:57:09.284958 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:09.285228 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:09.285280 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:09.289307 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:09.289363 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:09.289390 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:09.289412 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:09.289433 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:09.292006 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } attempt: 2
I20260620 08:57:09.292843 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.292536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58888)
0620 08:57:09.292703 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.292765 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 08:57:09.293529 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.293328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58896)
0620 08:57:09.293434 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.293475 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:57:09.293633 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34571: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
I20260620 08:57:09.295385 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33747,127.0.98.61:37755,127.0.98.60:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:09.296486 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:09.296664 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } attempt: 2
I20260620 08:57:09.297282 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.297035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58904)
0620 08:57:09.297158 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.297216 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 08:57:09.297732 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:57:09.297878 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.297675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58916)
0620 08:57:09.297768 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.297813 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:57:09.297986 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34571: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34571: connect: Connection refused (error 111)
W20260620 08:57:09.305955 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:09.305955 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:09.306087 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:09.306167 13865 server_base.cc:1034] running on GCE node
I20260620 08:57:09.306797 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:09.307912 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:09.309075 13865 hybrid_clock.cc:648] HybridClock initialized: now 1781945829309053 us; error 34 us; skew 500 ppm
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:09.311880 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 08:57:09.312709 13865 webserver.cc:460] Webserver started at http://127.0.98.60:46593/ using document root <none> and password file <none>
I20260620 08:57:09.313218 13865 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:09.313280 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:09.313453 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:09.315461 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data/instance:
uuid: "10b203b58489465796f58a3a4d10d9c8"
format_stamp: "Formatted at 2026-06-20 08:57:09 on dist-test-slave-fb1v"
server_key: "9147a015ca859dc18dab525208f96196"
server_key_iv: "57656578ec67a1e92ebd457602387643"
server_key_version: "encryptionkey@0"
I20260620 08:57:09.316047 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal/instance:
uuid: "10b203b58489465796f58a3a4d10d9c8"
format_stamp: "Formatted at 2026-06-20 08:57:09 on dist-test-slave-fb1v"
server_key: "9147a015ca859dc18dab525208f96196"
server_key_iv: "57656578ec67a1e92ebd457602387643"
server_key_version: "encryptionkey@0"
I20260620 08:57:09.320102 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 08:57:09.323143 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:09.324499 13865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:57:09.324630 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal
uuid: "10b203b58489465796f58a3a4d10d9c8"
format_stamp: "Formatted at 2026-06-20 08:57:09 on dist-test-slave-fb1v"
server_key: "9147a015ca859dc18dab525208f96196"
server_key_iv: "57656578ec67a1e92ebd457602387643"
server_key_version: "encryptionkey@0"
I20260620 08:57:09.324726 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:09.344794 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:09.348107 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:09.348320 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:09.356256 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34571
I20260620 08:57:09.356266 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34571 every 8 connection(s)
I20260620 08:57:09.357160 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
I20260620 08:57:09.359828 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:09.362641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 13865
I20260620 08:57:09.362735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945671001899-392-0/minicluster-data/master-2/wal/instance
I20260620 08:57:09.362993 392 external_mini_cluster.cc:1428] Setting key bb6d8a3fe0afb7eba781787822d34bbc
I20260620 08:57:09.363462 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 08:57:09.374483 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.365332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.0.98.62:33747)
0620 08:57:09.365444 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.365449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.365740 (+ 291us) server_negotiation.cc:407] Connection header received
0620 08:57:09.366031 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.366034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.366084 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.366148 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.366900 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.367389 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.368382 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.368587 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.368674 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.371057 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.371061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.371064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.371092 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.372431 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.372934 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.372937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.372938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.373007 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.373329 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.373332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.373333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.373566 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.373669 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.374197 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.374293 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:57:09.375388 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.365285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33747 (local address 127.0.0.1:42174)
0620 08:57:09.365609 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.365646 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.365872 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.366206 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.366212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.366533 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.366771 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.366783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.367483 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.367486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.368254 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.368263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.368717 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.368721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.368870 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.369417 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.369437 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.370868 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.372516 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.372524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.372535 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.372848 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.373087 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.373090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.373092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.373247 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.373657 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.373662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.373944 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.374136 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.374451 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260620 08:57:09.375401 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.364301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.0.98.60:34571)
0620 08:57:09.364727 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.364740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.364772 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:57:09.365575 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.365611 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.365885 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.366081 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.366837 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.367716 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.368500 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.368838 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.368973 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.371528 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.371534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.371543 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.371564 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.373204 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.373657 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.373662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.373664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.373744 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.373985 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.373988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.373990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.374255 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.374437 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.374804 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.374928 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260620 08:57:09.376986 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:09.384802 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.377535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.0.98.61:37755)
0620 08:57:09.377650 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.377654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.377781 (+ 127us) server_negotiation.cc:407] Connection header received
0620 08:57:09.377883 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.377886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.377934 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.378012 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.378341 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.378892 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.379661 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.379900 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.379984 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.381791 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.381794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.381796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.381818 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.383172 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.383571 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.383574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.383575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.383674 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.384000 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.384007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.384009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.384248 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.384338 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.384587 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.384695 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:57:09.384793 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.377460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37755 (local address 127.0.0.1:59564)
0620 08:57:09.377702 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.377714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.377797 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.378030 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.378032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.378157 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.378243 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.378248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.378987 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.378990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.379542 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.379550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.379998 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.380001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.380092 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.380524 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.380535 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.381661 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.383274 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.383277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.383279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.383490 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.383767 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.383770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.383772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.383881 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.384334 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.384337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.384426 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.384514 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.384618 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 08:57:09.385422 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:57:09.385612 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.377520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.62:33747)
0620 08:57:09.377620 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.377624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.377678 (+ 54us) server_negotiation.cc:407] Connection header received
0620 08:57:09.377767 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.377770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.377810 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.377879 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.378519 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.378986 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.379698 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.379898 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.379982 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.382290 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.382293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.382294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.382315 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.383833 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.384280 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.384282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.384284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.384364 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.384621 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.384624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.384625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.384816 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.385084 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.385369 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.385471 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:57:09.389133 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } attempt: 3
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:57:09.392791 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } attempt: 3
I20260620 08:57:09.393595 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.385919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58932)
0620 08:57:09.386138 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.386152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.386226 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.386396 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.386399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.386547 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.386623 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.386629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.387370 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.387373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.387931 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.387941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.388343 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.388347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.388424 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.388873 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.388886 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.390154 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.392329 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.392332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.392334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.392505 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.392760 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.392762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.392764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.392842 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.393192 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.393195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.393259 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.393336 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.393445 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260620 08:57:09.394027 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.386035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.0.98.60:34571)
0620 08:57:09.386139 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.386143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.386190 (+ 47us) server_negotiation.cc:407] Connection header received
0620 08:57:09.386288 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.386290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.386323 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.386379 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.386714 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.387271 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.388030 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.388254 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.388324 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.390266 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.390270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.390273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.390294 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.392237 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.392589 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.392592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.392594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.392670 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.392923 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.392925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.392927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.393111 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.393741 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.393803 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.393896 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 08:57:09.395525 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.386587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.0.98.61:37755)
0620 08:57:09.386681 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.386684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.386697 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:57:09.386763 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.386766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.386803 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.386879 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.387345 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.387861 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.388671 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.388943 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.389016 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.391379 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.391383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.391385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.391416 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.393206 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.393640 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.393643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.393645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.393732 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.394227 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.394231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.394233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.394472 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.394665 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.395274 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.395387 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25,"spinlock_wait_cycles":4864}
I20260620 08:57:09.397188 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: Bootstrap starting.
Jun 20 08:57:09 dist-test-slave-fb1v krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 08:57:09.399226 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.389676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58940)
0620 08:57:09.389781 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.389794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.389864 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.390585 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.390588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.390735 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.390819 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.390824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.392286 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.392290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.392949 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.392958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.393774 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.393777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.393854 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.394236 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.394261 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.395600 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.397820 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.397823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.397825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.398049 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.398354 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.398357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.398359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.398438 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.398790 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.398793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.398880 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.398959 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.399063 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"spinlock_wait_cycles":7936}
I20260620 08:57:09.399592 13934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:09.400328 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.389835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.0.98.60:34571)
0620 08:57:09.390397 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.390406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.390417 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:57:09.390445 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.390448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.390483 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.390556 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.390918 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.392153 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.393067 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.393652 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.394030 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.395745 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.395748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.395750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.395773 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.397276 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.398176 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.398179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.398181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.398256 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.398524 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.398526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.398527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.398689 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.399476 (+ 787us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.399542 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.399669 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":102,"threads_started":1}
I20260620 08:57:09.401297 13934 log.cc:824] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:09.402256 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.393138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34571 (local address 127.0.0.1:58952)
0620 08:57:09.393377 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:09.393391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:09.393457 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:09.394368 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:09.394371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:09.394528 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:09.394618 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.394625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.395676 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.395680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.396403 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:09.396415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.396952 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.396956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:09.397041 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.397428 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:09.397441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:09.398644 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:09.400730 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.400733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.400735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.400960 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.401261 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:09.401264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:09.401266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:09.401363 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:09.401789 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:09.401792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:09.401882 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 08:57:09.401972 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:57:09.402098 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 08:57:09.402881 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:57:09.402989 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.393358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.0.98.60:34571)
0620 08:57:09.394167 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.394170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.394181 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:57:09.394211 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.394213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.394246 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.394723 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.394826 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.395527 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.396543 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.396827 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.397244 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.398768 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.398771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.398774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.398793 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.400594 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.401074 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.401077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.401078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.401152 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.401473 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.401476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.401478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.401670 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.401792 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.402704 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.402808 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":96,"threads_started":1}
I20260620 08:57:09.403096 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6: Bootstrap starting.
I20260620 08:57:09.403661 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: No bootstrap required, opened a new log
I20260620 08:57:09.405704 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:09.405975 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: Bootstrap starting.
I20260620 08:57:09.406512 13859 log.cc:824] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:09.407526 13934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.407793 13934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:09.407837 13934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b203b58489465796f58a3a4d10d9c8, State: Initialized, Role: FOLLOWER
I20260620 08:57:09.408303 13934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.408502 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:09.408677 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6: No bootstrap required, opened a new log
I20260620 08:57:09.409548 13934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:09.409578 13784 log.cc:824] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:09.411995 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: No bootstrap required, opened a new log
W20260620 08:57:09.414665 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:57:09.414740 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:57:09.414955 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.415120 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.415800 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.405185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.0.98.61:37755)
0620 08:57:09.405291 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.405296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.405949 (+ 653us) server_negotiation.cc:407] Connection header received
0620 08:57:09.406183 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.406186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.406231 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.406280 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.407177 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.407895 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.409994 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.410297 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.410372 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.412315 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.412319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.412322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.412348 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.413882 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.414409 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.414412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.414414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.414497 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.414798 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.414801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.414803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.415011 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.415122 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.415500 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.415648 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:57:09.412855 13865 master_runner.cc:186] Error getting master registration for 127.0.98.62:33747: OK, instance_id { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" instance_seqno: 1781945828930681 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:57:09.416337 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:57:09.416345 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.416574 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:09.416630 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba9c8b6afaa4027bc995c20fefa8bb8, State: Initialized, Role: FOLLOWER
I20260620 08:57:09.417073 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.417268 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.417466 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:09.417518 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45702cd90008459695d62f4b002a7fd6, State: Initialized, Role: FOLLOWER
I20260620 08:57:09.417680 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.417939 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.417918 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.418354 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.418437 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:09.418527 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.419121 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 08:57:09.423620 13981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:57:09.423775 13981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:57:09.427490 13791 master_runner.cc:263] No leader master found from master 45702cd90008459695d62f4b002a7fd6
I20260620 08:57:09.427569 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:57:09.427731 13737 tablet.cc:2369] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:09.428069 13716 master_runner.cc:263] No leader master found from master 1ba9c8b6afaa4027bc995c20fefa8bb8
I20260620 08:57:09.428124 13716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:57:09.429396 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.409657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.0.98.60:34571)
0620 08:57:09.410098 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.410102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.420649 (+ 10547us) server_negotiation.cc:407] Connection header received
0620 08:57:09.420879 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.420883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.420930 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.421047 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.421947 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.422599 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.423569 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.423909 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.423976 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.425597 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.425601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.425604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.425628 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.427170 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.428009 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.428012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.428014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.428121 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.428556 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.428559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.428561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.428779 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.428904 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.429137 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.429251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
W20260620 08:57:09.434755 13987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:57:09.434906 13987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:57:09.436093 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:09.409322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.98.62:33747)
0620 08:57:09.409417 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:09.409422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:09.417380 (+ 7958us) server_negotiation.cc:407] Connection header received
0620 08:57:09.417541 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:09.417544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:09.417589 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:09.417637 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:09.418489 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.419203 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.420320 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:09.420616 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:09.420685 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:09.432595 (+ 11910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:09.432599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:09.432601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:09.432629 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:09.434208 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.434696 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.434700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.434702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.434786 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:09.435118 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:09.435122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:09.435124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:09.435364 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:09.435507 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:09.435808 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 08:57:09.435934 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:57:09.488853 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:09.489048 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.490103 13965 leader_election.cc:288] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ba9c8b6afaa4027bc995c20fefa8bb8 (127.0.98.62:33747), 10b203b58489465796f58a3a4d10d9c8 (127.0.98.60:34571)
I20260620 08:57:09.491106 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45702cd90008459695d62f4b002a7fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" is_pre_election: true
I20260620 08:57:09.491425 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45702cd90008459695d62f4b002a7fd6 in term 0.
I20260620 08:57:09.491845 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45702cd90008459695d62f4b002a7fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b203b58489465796f58a3a4d10d9c8" is_pre_election: true
I20260620 08:57:09.492136 13906 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45702cd90008459695d62f4b002a7fd6 in term 0.
I20260620 08:57:09.492617 13807 leader_election.cc:302] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b203b58489465796f58a3a4d10d9c8, 45702cd90008459695d62f4b002a7fd6; no voters:
I20260620 08:57:09.493101 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:57:09.493185 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:57:09.493239 13965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:09.494975 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.495333 13965 leader_election.cc:288] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ba9c8b6afaa4027bc995c20fefa8bb8 (127.0.98.62:33747), 10b203b58489465796f58a3a4d10d9c8 (127.0.98.60:34571)
I20260620 08:57:09.495934 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45702cd90008459695d62f4b002a7fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b203b58489465796f58a3a4d10d9c8"
I20260620 08:57:09.496052 13906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:09.497403 13906 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45702cd90008459695d62f4b002a7fd6 in term 1.
I20260620 08:57:09.497716 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45702cd90008459695d62f4b002a7fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8"
I20260620 08:57:09.497869 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:09.499039 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45702cd90008459695d62f4b002a7fd6 in term 1.
I20260620 08:57:09.499337 13807 leader_election.cc:302] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba9c8b6afaa4027bc995c20fefa8bb8, 45702cd90008459695d62f4b002a7fd6; no voters:
I20260620 08:57:09.499665 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:09.499888 13965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 45702cd90008459695d62f4b002a7fd6, State: Running, Role: LEADER
I20260620 08:57:09.502404 13965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } }
I20260620 08:57:09.505116 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45702cd90008459695d62f4b002a7fd6. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.505224 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:09.505577 13992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:09.506467 13992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:09.513183 13906 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [term 1 FOLLOWER]: Refusing update from remote peer 45702cd90008459695d62f4b002a7fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:09.513823 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:09.520258 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45702cd90008459695d62f4b002a7fd6. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.520386 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.521973 13758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [term 1 FOLLOWER]: Refusing update from remote peer 45702cd90008459695d62f4b002a7fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:09.522464 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:09.526860 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45702cd90008459695d62f4b002a7fd6. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.526983 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.532295 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.532434 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:09.533660 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.533789 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.536358 13993 mvcc.cc:205] Tried to move back new op lower bound from 7298850117678587904 to 7298850117644951552. Current Snapshot: MvccSnapshot[applied={T|T < 7298850117678587904}]
I20260620 08:57:09.537230 13992 catalog_manager.cc:1347] Generated new cluster ID: ef21b9f4310e4a9cb83b18e4e5f9f60e
I20260620 08:57:09.537289 13992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:09.547595 13996 mvcc.cc:205] Tried to move back new op lower bound from 7298850117678587904 to 7298850117644951552. Current Snapshot: MvccSnapshot[applied={T|T < 7298850117678587904 or (T in {7298850117678587904})}]
I20260620 08:57:09.553392 13992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:57:09.554579 13992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:09.560829 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.560981 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:09.562405 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45702cd90008459695d62f4b002a7fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba9c8b6afaa4027bc995c20fefa8bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } peers { permanent_uuid: "45702cd90008459695d62f4b002a7fd6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37755 } } peers { permanent_uuid: "10b203b58489465796f58a3a4d10d9c8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34571 } } }
I20260620 08:57:09.562551 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:57:09.569907 13994 mvcc.cc:205] Tried to move back new op lower bound from 7298850117678587904 to 7298850117644951552. Current Snapshot: MvccSnapshot[applied={T|T < 7298850117678587904}]
I20260620 08:57:09.586620 13992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45702cd90008459695d62f4b002a7fd6: Generated new TSK 0
I20260620 08:57:09.587395 13992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:57:10.416332 13962 catalog_manager.cc:1259] Loaded cluster ID: ef21b9f4310e4a9cb83b18e4e5f9f60e
I20260620 08:57:10.416416 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: loading cluster ID for follower catalog manager: success
I20260620 08:57:10.418112 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: acquiring CA information for follower catalog manager: success
I20260620 08:57:10.419286 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10b203b58489465796f58a3a4d10d9c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:57:10.424923 13981 catalog_manager.cc:1259] Loaded cluster ID: ef21b9f4310e4a9cb83b18e4e5f9f60e
I20260620 08:57:10.424988 13981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: loading cluster ID for follower catalog manager: success
I20260620 08:57:10.426518 13981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: acquiring CA information for follower catalog manager: success
I20260620 08:57:10.427644 13981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ba9c8b6afaa4027bc995c20fefa8bb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:57:10.658056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13716
I20260620 08:57:10.663254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13791
I20260620 08:57:10.668313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 13865
2026-06-20T08:57:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2989 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:57:10 dist-test-slave-fb1v krb5kdc[14007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:57:10 dist-test-slave-fb1v krb5kdc[14007](info): set up 2 sockets
Jun 20 08:57:10 dist-test-slave-fb1v krb5kdc[14007](info): commencing operation
krb5kdc: starting...
W20260620 08:57:11.631568 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:57:11 dist-test-slave-fb1v krb5kdc[14007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:57:11.754817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:11.864724 14020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:11.865008 14020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:11.865056 14020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:11.869076 14020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:11.869140 14020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:11.869159 14020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:11.869174 14020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:11.869189 14020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:11.874625 14020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:11.875720 14020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:11.876691 14020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:11.886027 14026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:11.886078 14028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:11.886039 14025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:11.886096 14020 server_base.cc:1034] running on GCE node
I20260620 08:57:11.886858 14020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:11.888002 14020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:11.889178 14020 hybrid_clock.cc:648] HybridClock initialized: now 1781945831889142 us; error 45 us; skew 500 ppm
Jun 20 08:57:11 dist-test-slave-fb1v krb5kdc[14007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:11.891804 14020 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:11.892741 14020 webserver.cc:460] Webserver started at http://127.0.98.62:44909/ using document root <none> and password file <none>
I20260620 08:57:11.893258 14020 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:11.893314 14020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:11.893473 14020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:11.895565 14020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "aecf62fab9b44fee9019b0b98981baad"
format_stamp: "Formatted at 2026-06-20 08:57:11 on dist-test-slave-fb1v"
server_key: "643c49f0bc4f35b6b78efff86c8db0ff"
server_key_iv: "b85600b558ed5817691903d97be14e3c"
server_key_version: "encryptionkey@0"
I20260620 08:57:11.896114 14020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "aecf62fab9b44fee9019b0b98981baad"
format_stamp: "Formatted at 2026-06-20 08:57:11 on dist-test-slave-fb1v"
server_key: "643c49f0bc4f35b6b78efff86c8db0ff"
server_key_iv: "b85600b558ed5817691903d97be14e3c"
server_key_version: "encryptionkey@0"
I20260620 08:57:11.900161 14020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 08:57:11.903784 14035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:11.905263 14020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:57:11.905405 14020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "aecf62fab9b44fee9019b0b98981baad"
format_stamp: "Formatted at 2026-06-20 08:57:11 on dist-test-slave-fb1v"
server_key: "643c49f0bc4f35b6b78efff86c8db0ff"
server_key_iv: "b85600b558ed5817691903d97be14e3c"
server_key_version: "encryptionkey@0"
I20260620 08:57:11.905524 14020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:11.924878 14020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:11.928278 14020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:11.928458 14020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:11.935393 14020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36289
I20260620 08:57:11.935438 14087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36289 every 8 connection(s)
I20260620 08:57:11.936362 14020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:11.939469 14088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:11.941716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14020
I20260620 08:57:11.941833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:57:11.942098 392 external_mini_cluster.cc:1428] Setting key 4e1663da96651f9c9da4d5d246a79ad5
I20260620 08:57:11.945741 14088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad: Bootstrap starting.
I20260620 08:57:11.947961 14088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:11.948836 14088 log.cc:824] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:11.951242 14088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad: No bootstrap required, opened a new log
Jun 20 08:57:11 dist-test-slave-fb1v krb5kdc[14007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:11.954646 14088 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf62fab9b44fee9019b0b98981baad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36289 } }
I20260620 08:57:11.954877 14088 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:11.954943 14088 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecf62fab9b44fee9019b0b98981baad, State: Initialized, Role: FOLLOWER
I20260620 08:57:11.955350 14088 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf62fab9b44fee9019b0b98981baad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36289 } }
I20260620 08:57:11.955471 14088 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:11.955534 14088 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:11.955649 14088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:11.956657 14088 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf62fab9b44fee9019b0b98981baad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36289 } }
I20260620 08:57:11.956964 14088 leader_election.cc:302] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecf62fab9b44fee9019b0b98981baad; no voters:
I20260620 08:57:11.957314 14088 leader_election.cc:288] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:57:11.957418 14093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:11.957726 14093 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [term 1 LEADER]: Becoming Leader. State: Replica: aecf62fab9b44fee9019b0b98981baad, State: Running, Role: LEADER
I20260620 08:57:11.958122 14093 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf62fab9b44fee9019b0b98981baad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36289 } }
I20260620 08:57:11.958508 14088 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:11.958706 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:11.943525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.98.62:36289)
0620 08:57:11.944048 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:11.944063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:11.944100 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:57:11.945044 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:11.945081 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:11.945640 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:11.946205 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:11.946975 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:11.948006 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:11.949214 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:11.949977 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:11.950152 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:11.952851 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:11.952857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:11.952871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:11.952899 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:11.955497 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:11.956048 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:11.956053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:11.956057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:11.956182 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:11.956449 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:11.956452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:11.956453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:11.956913 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:11.957124 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:11.957386 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 08:57:11.957707 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":152,"threads_started":1}
I20260620 08:57:11.959378 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecf62fab9b44fee9019b0b98981baad. Latest consensus state: current_term: 1 leader_uuid: "aecf62fab9b44fee9019b0b98981baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf62fab9b44fee9019b0b98981baad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36289 } } }
I20260620 08:57:11.959522 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:11.959854 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecf62fab9b44fee9019b0b98981baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf62fab9b44fee9019b0b98981baad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36289 } } }
I20260620 08:57:11.959983 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:11.959991 14102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:11.961776 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:57:11.963240 14102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:11.969084 14102 catalog_manager.cc:1347] Generated new cluster ID: 1af1cc9cfc15463098cfad834645f857
I20260620 08:57:11.969188 14102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:11.985958 14102 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:57:11.986943 14102 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:11.997582 14102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecf62fab9b44fee9019b0b98981baad: Generated new TSK 0
I20260620 08:57:11.998286 14102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:57:12.018612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14020
2026-06-20T08:57:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1350 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:57:12 dist-test-slave-fb1v krb5kdc[14115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:57:12 dist-test-slave-fb1v krb5kdc[14115](info): set up 2 sockets
Jun 20 08:57:12 dist-test-slave-fb1v krb5kdc[14115](info): commencing operation
krb5kdc: starting...
W20260620 08:57:12.979874 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:57:13.094509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:13.202222 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:13.202512 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:13.202569 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:13.206964 14128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:13.207059 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:13.207090 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:13.207113 14128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:13.207132 14128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:13.212847 14128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:13.214484 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:13.216176 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:13.225835 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.225869 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.225835 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:13.226049 14128 server_base.cc:1034] running on GCE node
I20260620 08:57:13.226680 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:13.227845 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:13.229038 14128 hybrid_clock.cc:648] HybridClock initialized: now 1781945833229014 us; error 39 us; skew 500 ppm
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:13.235533 14128 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:13.236517 14128 webserver.cc:460] Webserver started at http://127.0.98.62:37059/ using document root <none> and password file <none>
I20260620 08:57:13.237046 14128 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:13.237100 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:13.237263 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:13.239245 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "bd7b0271783d4ba0bbed93fb140f678b"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "1ad8bf28bbb75d7a5aba85ef072c76b3"
server_key_iv: "73558f2dccc0101c1c39921375779016"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.239794 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd7b0271783d4ba0bbed93fb140f678b"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "1ad8bf28bbb75d7a5aba85ef072c76b3"
server_key_iv: "73558f2dccc0101c1c39921375779016"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.243260 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:57:13.245694 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.246852 14128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:13.246953 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "bd7b0271783d4ba0bbed93fb140f678b"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "1ad8bf28bbb75d7a5aba85ef072c76b3"
server_key_iv: "73558f2dccc0101c1c39921375779016"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.247056 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:13.278354 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:13.281653 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:13.281839 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:13.288703 14128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41681
I20260620 08:57:13.288956 14195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41681 every 8 connection(s)
I20260620 08:57:13.289624 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:13.290540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14128
I20260620 08:57:13.290650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:57:13.290915 392 external_mini_cluster.cc:1428] Setting key 30f29502919d77507090afc52d065c99
I20260620 08:57:13.292560 14196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:13.299290 14196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap starting.
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:13.301687 14196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:13.302549 14196 log.cc:824] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:13.304812 14196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: No bootstrap required, opened a new log
I20260620 08:57:13.305922 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.292316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.0.98.62:41681)
0620 08:57:13.292725 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:13.292744 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:13.292783 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:57:13.293603 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:13.293624 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:13.294053 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:13.294397 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:13.295064 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.296619 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.297524 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.298206 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.298367 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.300828 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:13.300835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:13.300849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:13.300880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:13.302730 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.303279 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.303285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.303290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.303428 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.303758 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.303761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.303791 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.304283 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:13.304468 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:13.304699 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 08:57:13.305005 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260620 08:57:13.308732 14196 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:13.308933 14196 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:13.308972 14196 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7b0271783d4ba0bbed93fb140f678b, State: Initialized, Role: FOLLOWER
I20260620 08:57:13.309353 14196 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:13.309490 14196 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:13.309556 14196 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:13.309653 14196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:13.310513 14196 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:13.310815 14196 leader_election.cc:302] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7b0271783d4ba0bbed93fb140f678b; no voters:
I20260620 08:57:13.311084 14196 leader_election.cc:288] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:57:13.311179 14201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:13.311393 14201 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 LEADER]: Becoming Leader. State: Replica: bd7b0271783d4ba0bbed93fb140f678b, State: Running, Role: LEADER
I20260620 08:57:13.311750 14201 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:13.312234 14196 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:13.313340 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7b0271783d4ba0bbed93fb140f678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } } }
I20260620 08:57:13.313457 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:13.313337 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7b0271783d4ba0bbed93fb140f678b. Latest consensus state: current_term: 1 leader_uuid: "bd7b0271783d4ba0bbed93fb140f678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } } }
I20260620 08:57:13.313792 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:13.314378 14210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:13.316852 14210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:13.322012 14210 catalog_manager.cc:1347] Generated new cluster ID: eef9eaaab18b49caa22e8ef2858b8af5
I20260620 08:57:13.322089 14210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:13.337052 14210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:57:13.338191 14210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:13.344964 14210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Generated new TSK 0
I20260620 08:57:13.345718 14210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:57:13.382568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41681
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:13.489367 14222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:13.489605 14222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:13.489646 14222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:13.493546 14222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:13.493597 14222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:13.493656 14222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:13.498682 14222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:13.499694 14222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:13.500658 14222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:13.509514 14227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.509516 14230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.509614 14228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:13.509709 14222 server_base.cc:1034] running on GCE node
I20260620 08:57:13.510064 14222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:13.510627 14222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:13.511814 14222 hybrid_clock.cc:648] HybridClock initialized: now 1781945833511799 us; error 26 us; skew 500 ppm
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:13.514411 14222 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:13.515184 14222 webserver.cc:460] Webserver started at http://127.0.98.1:44521/ using document root <none> and password file <none>
I20260620 08:57:13.515718 14222 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:13.515784 14222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:13.515969 14222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:13.517856 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "8628b8a778934989b74758dc4cabf5d7"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "172b177d103a224215bd0a74660751fc"
server_key_iv: "c5ffe720e0424dc02a41725054c66e83"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.518392 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8628b8a778934989b74758dc4cabf5d7"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "172b177d103a224215bd0a74660751fc"
server_key_iv: "c5ffe720e0424dc02a41725054c66e83"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.521760 14222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:57:13.524060 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.525200 14222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:13.525315 14222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "8628b8a778934989b74758dc4cabf5d7"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "172b177d103a224215bd0a74660751fc"
server_key_iv: "c5ffe720e0424dc02a41725054c66e83"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.525403 14222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:13.546572 14222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:13.549279 14222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:13.549438 14222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:13.550004 14222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:13.550833 14222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:13.550891 14222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.550931 14222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:13.550951 14222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.558737 14222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43473
I20260620 08:57:13.558781 14350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43473 every 8 connection(s)
I20260620 08:57:13.559520 14222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:13.568758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14222
I20260620 08:57:13.568874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:57:13.569188 392 external_mini_cluster.cc:1428] Setting key 3d013d573a1008683f97205e4c2d7bd6
I20260620 08:57:13.572475 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.561174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59815 (local address 127.0.98.62:41681)
0620 08:57:13.561310 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:13.561314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:13.562200 (+ 886us) server_negotiation.cc:407] Connection header received
0620 08:57:13.563035 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:13.563038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:13.563093 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:13.563174 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:13.564320 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.564846 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.565924 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.566175 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.566261 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.568572 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:13.568576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:13.568579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:13.568603 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:13.570243 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.570842 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.570845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.570846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.570929 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.571305 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.571308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.571310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.571490 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:13.571575 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:13.572175 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 08:57:13.572313 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 08:57:13.573264 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.561448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:59815)
0620 08:57:13.562045 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:13.562083 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:13.562846 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:13.563334 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:13.563341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:13.563714 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:13.564177 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:13.564191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.564961 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.564964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:13.565805 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:13.565814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.566308 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.566312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:13.566461 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.566973 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:13.566989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:13.568410 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:13.570372 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:13.570377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:13.570387 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:13.570744 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:13.571018 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:13.571021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:13.571024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:13.571194 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:13.571586 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:13.571592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:13.571921 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 08:57:13.572112 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 08:57:13.572459 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":102,"threads_started":1}
I20260620 08:57:13.574445 14351 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41681
I20260620 08:57:13.574679 14351 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:13.575145 14351 heartbeater.cc:511] Master 127.0.98.62:41681 requested a full tablet report, sending...
I20260620 08:57:13.576681 14160 ts_manager.cc:194] Registered new tserver with Master: 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473)
I20260620 08:57:13.578178 14160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59815
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:57:13.594556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41681
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:13.701684 14356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:13.701934 14356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:13.701977 14356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:13.705916 14356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:13.705966 14356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:13.706022 14356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:13.711439 14356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:13.712469 14356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:13.713449 14356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:57:13.723363 14356 server_base.cc:1034] running on GCE node
W20260620 08:57:13.723311 14361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.723377 14362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.723313 14364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:13.724037 14356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:13.724674 14356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:13.725852 14356 hybrid_clock.cc:648] HybridClock initialized: now 1781945833725833 us; error 32 us; skew 500 ppm
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:13.728711 14356 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:13.729665 14356 webserver.cc:460] Webserver started at http://127.0.98.2:46005/ using document root <none> and password file <none>
I20260620 08:57:13.730196 14356 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:13.730252 14356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:13.730409 14356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:13.732416 14356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "44fc5e5fe6a24f898dc6c0ae2e326845"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "5ee8dac6f4d1c24f6433bb2051de39fe"
server_key_iv: "73c25a9115804e353df6e4039bd4da57"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.732947 14356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44fc5e5fe6a24f898dc6c0ae2e326845"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "5ee8dac6f4d1c24f6433bb2051de39fe"
server_key_iv: "73c25a9115804e353df6e4039bd4da57"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.737057 14356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 08:57:13.740248 14371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.741837 14356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:57:13.741982 14356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "44fc5e5fe6a24f898dc6c0ae2e326845"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "5ee8dac6f4d1c24f6433bb2051de39fe"
server_key_iv: "73c25a9115804e353df6e4039bd4da57"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.742087 14356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:13.755052 14356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:13.758103 14356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:13.758286 14356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:13.758826 14356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:13.759639 14356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:13.759697 14356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.759744 14356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:13.759765 14356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.767932 14356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38729
I20260620 08:57:13.768018 14484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38729 every 8 connection(s)
I20260620 08:57:13.768775 14356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:13.770373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14356
I20260620 08:57:13.770481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:57:13.770783 392 external_mini_cluster.cc:1428] Setting key 74c2f0ecdefbe8654e19910a7bf413d4
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:57:13.782853 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.770535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47167 (local address 127.0.98.62:41681)
0620 08:57:13.770718 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:13.770723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:13.771799 (+ 1076us) server_negotiation.cc:407] Connection header received
0620 08:57:13.772822 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:13.772825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:13.772881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:13.772951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:13.774292 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.774811 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.776457 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.776682 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.776764 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.779428 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:13.779431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:13.779433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:13.779456 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:13.780830 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.781297 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.781301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.781303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.781378 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.781726 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.781730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.781732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.781968 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:13.782065 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:13.782614 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 08:57:13.782719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 08:57:13.783885 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.770932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:47167)
0620 08:57:13.771664 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:13.771719 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:13.772622 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:13.773124 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:13.773131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:13.773483 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:13.774115 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:13.774134 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.774930 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.774934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:13.776327 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:13.776339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.776834 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.776838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:13.777039 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.777664 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:13.777686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:13.779269 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:13.780937 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:13.780942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:13.780956 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:13.781204 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:13.781460 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:13.781463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:13.781465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:13.781639 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:13.782066 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:13.782073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:13.782340 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 08:57:13.782545 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 08:57:13.782881 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":125,"threads_started":1}
I20260620 08:57:13.784965 14485 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41681
I20260620 08:57:13.785177 14485 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:13.785647 14485 heartbeater.cc:511] Master 127.0.98.62:41681 requested a full tablet report, sending...
I20260620 08:57:13.786621 14160 ts_manager.cc:194] Registered new tserver with Master: 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729)
I20260620 08:57:13.787393 14160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:57:13.796113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41681
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:13.904178 14490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:13.904433 14490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:13.904475 14490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:13.908396 14490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:13.908445 14490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:13.908501 14490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:13.913792 14490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:13.914808 14490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:13.915827 14490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:57:13.925750 14490 server_base.cc:1034] running on GCE node
W20260620 08:57:13.925714 14498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.925712 14495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:13.925830 14496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:13.926362 14490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:13.926988 14490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:13.928165 14490 hybrid_clock.cc:648] HybridClock initialized: now 1781945833928141 us; error 39 us; skew 500 ppm
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:13.931026 14490 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:13.931931 14490 webserver.cc:460] Webserver started at http://127.0.98.3:41779/ using document root <none> and password file <none>
I20260620 08:57:13.932462 14490 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:13.932530 14490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:13.932721 14490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:13.934821 14490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "30fc4254eb3647708c286246691fe973"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "0185ecb1ea2104cf70b10d614bcfd918"
server_key_iv: "3f845d7b6f105d602ada3ef11dee2059"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.935366 14490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30fc4254eb3647708c286246691fe973"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "0185ecb1ea2104cf70b10d614bcfd918"
server_key_iv: "3f845d7b6f105d602ada3ef11dee2059"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.938831 14490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:57:13.941111 14505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.942348 14490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:57:13.942453 14490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "30fc4254eb3647708c286246691fe973"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "0185ecb1ea2104cf70b10d614bcfd918"
server_key_iv: "3f845d7b6f105d602ada3ef11dee2059"
server_key_version: "encryptionkey@0"
I20260620 08:57:13.942543 14490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:13.969471 14490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:13.972457 14490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:13.972642 14490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:13.973237 14490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:13.974040 14490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:13.974085 14490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.974135 14490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:13.974161 14490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:13.982848 14490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42657
I20260620 08:57:13.982919 14618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42657 every 8 connection(s)
I20260620 08:57:13.983805 14490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:57:13.992121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14490
I20260620 08:57:13.992228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:57:13.992488 392 external_mini_cluster.cc:1428] Setting key 2bafc69bc00b2ee55a9b274b61e5f332
Jun 20 08:57:13 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:13.997232 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.985583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35577 (local address 127.0.98.62:41681)
0620 08:57:13.985746 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:13.985749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:13.986884 (+ 1135us) server_negotiation.cc:407] Connection header received
0620 08:57:13.987780 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:13.987783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:13.987832 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:13.987906 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:13.989056 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.989586 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.990745 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.990969 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.991050 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.993553 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:13.993556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:13.993558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:13.993584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:13.995210 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.995782 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.995785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.995787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.995874 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:13.996249 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:13.996252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:13.996254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:13.996464 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:13.996544 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:13.997016 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 08:57:13.997114 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 08:57:13.998088 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:13.985894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:35577)
0620 08:57:13.986733 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:13.986775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:13.987566 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:13.988110 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:13.988117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:13.988456 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:13.988910 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:13.988924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.989736 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.989744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:13.990624 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:13.990634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:13.991121 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:13.991124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:13.991282 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:13.991823 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:13.991839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:13.993394 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:13.995349 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:13.995354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:13.995367 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:13.995673 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:13.995957 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:13.995959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:13.995961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:13.996111 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:13.996574 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:13.996579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:13.996821 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 08:57:13.996974 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 08:57:13.997251 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":513,"threads_started":1}
I20260620 08:57:13.999089 14619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41681
I20260620 08:57:13.999315 14619 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:13.999795 14619 heartbeater.cc:511] Master 127.0.98.62:41681 requested a full tablet report, sending...
I20260620 08:57:14.000795 14160 ts_manager.cc:194] Registered new tserver with Master: 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
I20260620 08:57:14.001348 14160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35577
I20260620 08:57:14.006829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:57:14.016777 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.008676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.0.98.62:41681)
0620 08:57:14.008792 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.008796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.008963 (+ 167us) server_negotiation.cc:407] Connection header received
0620 08:57:14.009039 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.009041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.009082 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.009140 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:14.009739 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.010319 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.011281 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.011506 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.011593 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.012976 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:14.012979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:14.012981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:14.013003 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:14.014693 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.015180 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.015185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.015187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.015364 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.015881 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.015884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.015885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.016120 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:14.016238 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.016553 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.016656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:57:14.020421 14160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:57:14.023365 14160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:57:14.036919 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.031856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.0.98.2:38729)
0620 08:57:14.032331 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.032343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.032372 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:57:14.032447 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.032452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.032647 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.032780 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.033040 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.034052 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.035532 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.036255 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.036358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.036498 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.036671 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.036775 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":135,"threads_started":1}
I20260620 08:57:14.036919 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.031398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38729 (local address 127.0.0.1:55224)
0620 08:57:14.032219 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.032232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.032312 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.032793 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.032798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.032820 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.032944 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.032950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.034211 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.034214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.035427 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.035435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.036369 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.036373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.036476 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.036515 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.036616 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.036719 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":81,"threads_started":1}
I20260620 08:57:14.037521 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.031311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43473 (local address 127.0.0.1:33304)
0620 08:57:14.031681 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.031730 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.031889 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.033815 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.033818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.033834 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.033927 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.033934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.035140 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.035144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.036122 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.036129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.037095 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.037099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.037191 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.037211 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.037304 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.037368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260620 08:57:14.037657 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.031426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.0.98.1:43473)
0620 08:57:14.033481 (+ 2055us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.033496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.033513 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:57:14.033570 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.033573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.033693 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.033816 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.034030 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.034996 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.036232 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.036969 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.037097 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.037226 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.037348 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.037439 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1956,"thread_start_us":101,"threads_started":1}
I20260620 08:57:14.038030 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.032444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.98.3:42657)
0620 08:57:14.032876 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.032884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.032900 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:57:14.032957 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.032960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.033080 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.033199 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.033432 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.034480 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.035909 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.036641 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.036729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.036848 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.037806 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.037899 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":93,"threads_started":1}
I20260620 08:57:14.038225 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.032049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42657 (local address 127.0.0.1:53188)
0620 08:57:14.032573 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.032586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.032662 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.033201 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.033203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.033219 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.033311 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.033318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.034607 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.034611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.035777 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.035787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.036905 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.036909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.036993 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.037407 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.037472 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.037537 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":79,"threads_started":1}
I20260620 08:57:14.040282 14285 tablet_service.cc:1467] Processing CreateTablet for tablet 43202d67804e44668c0cd338f6569b7e (DEFAULT_TABLE table=test-table [id=13a5bf451df2475998a9713ed69c9179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:14.040287 14419 tablet_service.cc:1467] Processing CreateTablet for tablet 43202d67804e44668c0cd338f6569b7e (DEFAULT_TABLE table=test-table [id=13a5bf451df2475998a9713ed69c9179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:14.041154 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43202d67804e44668c0cd338f6569b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:14.041154 14419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43202d67804e44668c0cd338f6569b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:14.046627 14553 tablet_service.cc:1467] Processing CreateTablet for tablet 43202d67804e44668c0cd338f6569b7e (DEFAULT_TABLE table=test-table [id=13a5bf451df2475998a9713ed69c9179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:14.047477 14637 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Bootstrap starting.
I20260620 08:57:14.048159 14636 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Bootstrap starting.
I20260620 08:57:14.048285 14553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43202d67804e44668c0cd338f6569b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:14.049818 14637 tablet_bootstrap.cc:654] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:14.050182 14636 tablet_bootstrap.cc:654] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:14.050575 14637 log.cc:824] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:14.051338 14636 log.cc:824] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:14.052484 14637 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: No bootstrap required, opened a new log
I20260620 08:57:14.052671 14637 ts_tablet_manager.cc:1397] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:57:14.053256 14636 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: No bootstrap required, opened a new log
I20260620 08:57:14.053428 14636 ts_tablet_manager.cc:1397] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 08:57:14.054761 14640 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Bootstrap starting.
I20260620 08:57:14.056568 14640 tablet_bootstrap.cc:654] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:14.056397 14637 raft_consensus.cc:348] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.056646 14637 raft_consensus.cc:374] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:14.056710 14637 raft_consensus.cc:729] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fc5e5fe6a24f898dc6c0ae2e326845, State: Initialized, Role: FOLLOWER
I20260620 08:57:14.057034 14636 raft_consensus.cc:348] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.057163 14637 consensus_queue.cc:260] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.057255 14636 raft_consensus.cc:374] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:14.057298 14636 raft_consensus.cc:729] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8628b8a778934989b74758dc4cabf5d7, State: Initialized, Role: FOLLOWER
I20260620 08:57:14.057672 14636 consensus_queue.cc:260] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.057787 14640 log.cc:824] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:14.058090 14637 ts_tablet_manager.cc:1428] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:57:14.058223 14485 heartbeater.cc:503] Master 127.0.98.62:41681 was elected leader, sending a full tablet report...
I20260620 08:57:14.058473 14636 ts_tablet_manager.cc:1428] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 08:57:14.058758 14351 heartbeater.cc:503] Master 127.0.98.62:41681 was elected leader, sending a full tablet report...
I20260620 08:57:14.060318 14640 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: No bootstrap required, opened a new log
I20260620 08:57:14.060480 14640 ts_tablet_manager.cc:1397] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260620 08:57:14.060505 14352 tablet.cc:2369] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:14.063851 14640 raft_consensus.cc:348] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.064090 14640 raft_consensus.cc:374] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:14.064144 14640 raft_consensus.cc:729] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fc4254eb3647708c286246691fe973, State: Initialized, Role: FOLLOWER
I20260620 08:57:14.064528 14640 consensus_queue.cc:260] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.065066 14619 heartbeater.cc:503] Master 127.0.98.62:41681 was elected leader, sending a full tablet report...
I20260620 08:57:14.065315 14640 ts_tablet_manager.cc:1428] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:57:14.108220 14644 raft_consensus.cc:482] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:14.108443 14644 raft_consensus.cc:504] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.109567 14644 leader_election.cc:288] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473), 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729)
I20260620 08:57:14.114291 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.109871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38729 (local address 127.0.98.3:60189)
0620 08:57:14.109988 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.110003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.110098 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.110492 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.110495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.110508 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.110631 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.110658 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.111525 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.111529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.112851 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.112862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.113742 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.113746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.113940 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.113979 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.114053 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.114130 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":32}
I20260620 08:57:14.114306 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.110168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35769 (local address 127.0.98.1:43473)
0620 08:57:14.110270 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.110275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.110501 (+ 226us) server_negotiation.cc:407] Connection header received
0620 08:57:14.110603 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.110605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.110650 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.110732 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.111064 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.111694 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.113085 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.113677 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.113774 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.113832 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.114087 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.114173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:57:14.114306 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.110111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60189 (local address 127.0.98.2:38729)
0620 08:57:14.110241 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.110246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.110264 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:57:14.110324 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.110327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.110380 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.110460 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.110785 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.111433 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.112976 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.113624 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.113749 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.113827 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.114107 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.114178 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:57:14.114302 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.110050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43473 (local address 127.0.98.3:35769)
0620 08:57:14.110403 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.110413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.110503 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.110823 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.110825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.110846 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.110946 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.110952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.111811 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.111814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.112981 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.112988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.113768 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.113782 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.113922 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.113983 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.114053 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.114130 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":61,"thread_start_us":147,"threads_started":1}
I20260620 08:57:14.114959 14305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43202d67804e44668c0cd338f6569b7e" candidate_uuid: "30fc4254eb3647708c286246691fe973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8628b8a778934989b74758dc4cabf5d7" is_pre_election: true
I20260620 08:57:14.114959 14439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43202d67804e44668c0cd338f6569b7e" candidate_uuid: "30fc4254eb3647708c286246691fe973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" is_pre_election: true
I20260620 08:57:14.115268 14439 raft_consensus.cc:2454] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fc4254eb3647708c286246691fe973 in term 0.
I20260620 08:57:14.115268 14305 raft_consensus.cc:2454] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fc4254eb3647708c286246691fe973 in term 0.
I20260620 08:57:14.115752 14509 leader_election.cc:302] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845; no voters:
I20260620 08:57:14.116030 14644 raft_consensus.cc:2781] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:57:14.116113 14644 raft_consensus.cc:482] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:57:14.116161 14644 raft_consensus.cc:3037] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:14.117168 14644 raft_consensus.cc:504] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.117487 14644 leader_election.cc:288] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [CANDIDATE]: Term 1 election: Requested vote from peers 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473), 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729)
I20260620 08:57:14.117812 14439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43202d67804e44668c0cd338f6569b7e" candidate_uuid: "30fc4254eb3647708c286246691fe973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845"
I20260620 08:57:14.117835 14305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43202d67804e44668c0cd338f6569b7e" candidate_uuid: "30fc4254eb3647708c286246691fe973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8628b8a778934989b74758dc4cabf5d7"
I20260620 08:57:14.117950 14305 raft_consensus.cc:3037] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:14.117938 14439 raft_consensus.cc:3037] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:14.118891 14439 raft_consensus.cc:2454] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fc4254eb3647708c286246691fe973 in term 1.
I20260620 08:57:14.118925 14305 raft_consensus.cc:2454] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fc4254eb3647708c286246691fe973 in term 1.
I20260620 08:57:14.119130 14509 leader_election.cc:302] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845; no voters:
I20260620 08:57:14.119297 14644 raft_consensus.cc:2781] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:14.119511 14644 raft_consensus.cc:686] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 1 LEADER]: Becoming Leader. State: Replica: 30fc4254eb3647708c286246691fe973, State: Running, Role: LEADER
I20260620 08:57:14.119884 14644 consensus_queue.cc:237] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.123070 14159 catalog_manager.cc:5495] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30fc4254eb3647708c286246691fe973 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "30fc4254eb3647708c286246691fe973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } health_report { overall_health: HEALTHY } } }
I20260620 08:57:14.138118 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.134806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.0.98.3:42657)
0620 08:57:14.134949 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.134953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.134967 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:57:14.135046 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.135049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.135104 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.135196 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:14.135422 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.135971 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.136770 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.137010 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.137111 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.137372 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:14.137757 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:57:14.137858 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.137921 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.137974 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 08:57:14.143426 14439 raft_consensus.cc:1261] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Refusing update from remote peer 30fc4254eb3647708c286246691fe973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:14.143426 14305 raft_consensus.cc:1261] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Refusing update from remote peer 30fc4254eb3647708c286246691fe973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:14.144156 14646 consensus_queue.cc:1035] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:14.144318 14644 consensus_queue.cc:1035] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:14.153335 14651 mvcc.cc:205] Tried to move back new op lower bound from 7298850136645099520 to 7298850136557838336. Current Snapshot: MvccSnapshot[applied={T|T < 7298850136645099520 or (T in {7298850136645099520})}]
I20260620 08:57:14.160391 14160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:14.160549 14160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:14.162751 14160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Sending DeleteTablet for 3 replicas of tablet 43202d67804e44668c0cd338f6569b7e
I20260620 08:57:14.163563 14419 tablet_service.cc:1514] Processing DeleteTablet for tablet 43202d67804e44668c0cd338f6569b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55224
I20260620 08:57:14.163530 14285 tablet_service.cc:1514] Processing DeleteTablet for tablet 43202d67804e44668c0cd338f6569b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33304
I20260620 08:57:14.163638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14222
I20260620 08:57:14.164573 14553 tablet_service.cc:1514] Processing DeleteTablet for tablet 43202d67804e44668c0cd338f6569b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260620 08:57:14.164636 14659 tablet_replica.cc:330] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: stopping tablet replica
I20260620 08:57:14.165007 14659 raft_consensus.cc:2229] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:14.165297 14659 raft_consensus.cc:2258] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:14.165493 14660 tablet_replica.cc:330] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: stopping tablet replica
I20260620 08:57:14.165802 14660 raft_consensus.cc:2229] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:57:14.166165 14660 raft_consensus.cc:2258] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:14.167495 14659 ts_tablet_manager.cc:1905] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:14.167637 14660 ts_tablet_manager.cc:1905] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:57:14.170208 14145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43473 (error 108)
I20260620 08:57:14.170248 14659 ts_tablet_manager.cc:1918] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:14.170349 14660 ts_tablet_manager.cc:1918] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:14.170441 14660 log.cc:1196] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/43202d67804e44668c0cd338f6569b7e
I20260620 08:57:14.170521 14659 log.cc:1196] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/43202d67804e44668c0cd338f6569b7e
I20260620 08:57:14.170744 14660 ts_tablet_manager.cc:1939] T 43202d67804e44668c0cd338f6569b7e P 30fc4254eb3647708c286246691fe973: Deleting consensus metadata
I20260620 08:57:14.170953 14659 ts_tablet_manager.cc:1939] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting consensus metadata
I20260620 08:57:14.170979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14356
I20260620 08:57:14.172858 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.172595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43473 (local address 127.0.0.1:33318)
0620 08:57:14.172707 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.172772 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:57:14.173290 14145 catalog_manager.cc:4584] TS 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43202d67804e44668c0cd338f6569b7e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43473: connect: Connection refused (error 111)
I20260620 08:57:14.174368 14146 catalog_manager.cc:4841] TS 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657): tablet 43202d67804e44668c0cd338f6569b7e (table test-table [id=13a5bf451df2475998a9713ed69c9179]) successfully deleted
I20260620 08:57:14.176872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14490
I20260620 08:57:14.177062 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.176808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38729 (local address 127.0.0.1:55236)
0620 08:57:14.176937 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.176988 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 08:57:14.181687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14128
I20260620 08:57:14.186795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41681
--webserver_interface=127.0.98.62
--webserver_port=37059
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:14.293977 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:14.294270 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:14.294312 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:14.298578 14662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:14.298660 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:14.298679 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:14.298695 14662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:14.298709 14662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:14.304922 14662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:14.305950 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:14.306937 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:14.319465 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.319497 14669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.319514 14671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:14.319726 14662 server_base.cc:1034] running on GCE node
I20260620 08:57:14.320256 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:14.321350 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:14.322553 14662 hybrid_clock.cc:648] HybridClock initialized: now 1781945834322523 us; error 42 us; skew 500 ppm
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:14.325338 14662 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:14.326246 14662 webserver.cc:460] Webserver started at http://127.0.98.62:37059/ using document root <none> and password file <none>
I20260620 08:57:14.326792 14662 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:14.326872 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:14.331074 14662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:57:14.333930 14678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.335222 14662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:57:14.335338 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "bd7b0271783d4ba0bbed93fb140f678b"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "1ad8bf28bbb75d7a5aba85ef072c76b3"
server_key_iv: "73558f2dccc0101c1c39921375779016"
server_key_version: "encryptionkey@0"
I20260620 08:57:14.336000 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:14.348419 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:14.351419 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:14.351596 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:14.358245 14662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41681
I20260620 08:57:14.358503 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41681 every 8 connection(s)
I20260620 08:57:14.359321 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:14.362766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14662
I20260620 08:57:14.363162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41681
--webserver_port=44521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:14.363646 14731 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:57:14.365639 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap starting.
I20260620 08:57:14.382134 14731 log.cc:824] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:14.390491 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:14.390846 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap complete.
I20260620 08:57:14.394680 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:14.395072 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7b0271783d4ba0bbed93fb140f678b, State: Initialized, Role: FOLLOWER
I20260620 08:57:14.395520 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:14.395661 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:14.395733 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:14.395826 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:57:14.397929 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:14.398249 14731 leader_election.cc:302] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7b0271783d4ba0bbed93fb140f678b; no voters:
I20260620 08:57:14.398535 14731 leader_election.cc:288] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:57:14.398636 14735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:57:14.398845 14735 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 LEADER]: Becoming Leader. State: Replica: bd7b0271783d4ba0bbed93fb140f678b, State: Running, Role: LEADER
I20260620 08:57:14.399129 14735 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:14.399797 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:14.400454 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bd7b0271783d4ba0bbed93fb140f678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } } }
I20260620 08:57:14.400583 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:14.400888 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7b0271783d4ba0bbed93fb140f678b. Latest consensus state: current_term: 2 leader_uuid: "bd7b0271783d4ba0bbed93fb140f678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } } }
I20260620 08:57:14.400967 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:14.401454 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:14.405989 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:14.406626 14742 catalog_manager.cc:1259] Loaded cluster ID: eef9eaaab18b49caa22e8ef2858b8af5
I20260620 08:57:14.406700 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:14.408156 14742 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 08:57:14.412938 14742 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 08:57:14.418545 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:14.419373 14742 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Loaded TSK: 0
I20260620 08:57:14.420595 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:57:14.478605 14733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:14.478860 14733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:14.478924 14733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:14.482961 14733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:14.483028 14733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:14.483119 14733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:14.488391 14733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44521
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:14.489356 14733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:14.490280 14733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:14.499342 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.499382 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.499342 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:14.499799 14733 server_base.cc:1034] running on GCE node
I20260620 08:57:14.500254 14733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:14.501039 14733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:14.502228 14733 hybrid_clock.cc:648] HybridClock initialized: now 1781945834502201 us; error 38 us; skew 500 ppm
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:14.505614 14733 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:14.506675 14733 webserver.cc:460] Webserver started at http://127.0.98.1:44521/ using document root <none> and password file <none>
I20260620 08:57:14.507364 14733 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:14.507459 14733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:14.512274 14733 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:57:14.514319 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.515554 14733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:14.515689 14733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "8628b8a778934989b74758dc4cabf5d7"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "172b177d103a224215bd0a74660751fc"
server_key_iv: "c5ffe720e0424dc02a41725054c66e83"
server_key_version: "encryptionkey@0"
I20260620 08:57:14.516193 14733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:14.534286 14733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:14.537209 14733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:14.537370 14733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:14.537941 14733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:14.538966 14775 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:57:14.543969 14733 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:57:14.544039 14733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 08:57:14.544106 14733 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:57:14.545724 14733 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:57:14.545770 14733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 08:57:14.546059 14775 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Bootstrap starting.
I20260620 08:57:14.554934 14733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43473
I20260620 08:57:14.555158 14882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43473 every 8 connection(s)
I20260620 08:57:14.555931 14733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:57:14.559723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14733
I20260620 08:57:14.560097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38729
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41681
--webserver_port=46005
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:14.564431 14775 log.cc:824] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:14.572270 14775 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:14.572650 14775 tablet_bootstrap.cc:492] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Bootstrap complete.
I20260620 08:57:14.573854 14775 ts_tablet_manager.cc:1397] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260620 08:57:14.574376 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.557938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55713 (local address 127.0.98.62:41681)
0620 08:57:14.558494 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.558514 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.558576 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:57:14.559677 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.559693 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.560088 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.560452 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:14.562495 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.563459 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.565073 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.565626 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.565809 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.568815 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:14.568828 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:14.568845 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:14.568880 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:14.570910 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.571532 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.571538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.571543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.571701 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.572092 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.572095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.572097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.572502 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:14.572623 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.573258 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.573537 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":154,"threads_started":1}
I20260620 08:57:14.574378 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.557876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:55713)
0620 08:57:14.558336 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.558387 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.559167 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.561060 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.561069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.561578 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:14.562283 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.562310 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.563644 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.563648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.564937 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.564948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.565776 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.565780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.565984 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.566682 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:14.566712 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:14.568604 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:14.571051 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.571058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.571069 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.571402 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.571798 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.571801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.571804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.571974 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.572621 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:14.572628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:14.572918 (+ 290us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.573134 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.573440 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260620 08:57:14.576536 14775 raft_consensus.cc:348] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.577023 14775 raft_consensus.cc:729] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8628b8a778934989b74758dc4cabf5d7, State: Initialized, Role: FOLLOWER
I20260620 08:57:14.577216 14883 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41681
I20260620 08:57:14.577416 14883 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:14.577447 14775 consensus_queue.cc:260] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } } peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } }
I20260620 08:57:14.577947 14883 heartbeater.cc:511] Master 127.0.98.62:41681 requested a full tablet report, sending...
I20260620 08:57:14.578435 14775 ts_tablet_manager.cc:1428] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:57:14.580448 14695 ts_manager.cc:194] Registered new tserver with Master: 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473)
I20260620 08:57:14.584236 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55713
I20260620 08:57:14.586410 14883 heartbeater.cc:503] Master 127.0.98.62:41681 was elected leader, sending a full tablet report...
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:57:14.594347 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.583546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43473 (local address 127.0.0.1:33332)
0620 08:57:14.584223 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.584255 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.584389 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.584819 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.584824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.585313 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:14.585437 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.585446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.586739 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.586742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.587366 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.587376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.588042 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.588058 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.588160 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.588636 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:14.588647 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:14.590670 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:14.592565 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.592573 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.592578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.592832 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.593114 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.593117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.593119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.593198 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.593718 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:14.593724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:14.593836 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.593964 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.594160 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":151,"threads_started":1}
I20260620 08:57:14.594347 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.583627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.0.98.1:43473)
0620 08:57:14.583974 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.583985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.584344 (+ 359us) server_negotiation.cc:407] Connection header received
0620 08:57:14.584532 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.584536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.584680 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.584802 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:14.585558 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.586624 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.587475 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.587948 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.588042 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.590922 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:14.590935 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:14.590948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:14.590972 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:14.592450 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.592941 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.592946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.592948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.593027 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.593280 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.593284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.593285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.593584 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:14.593722 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.594043 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.594189 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":71,"threads_started":1}
I20260620 08:57:14.595278 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 43202d67804e44668c0cd338f6569b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33332
I20260620 08:57:14.596695 14892 tablet_replica.cc:330] stopping tablet replica
I20260620 08:57:14.596961 14892 raft_consensus.cc:2229] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:14.597172 14892 raft_consensus.cc:2258] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:14.598024 14892 ts_tablet_manager.cc:1905] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:14.601460 14892 ts_tablet_manager.cc:1918] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:14.601694 14892 log.cc:1196] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/43202d67804e44668c0cd338f6569b7e
I20260620 08:57:14.602065 14892 ts_tablet_manager.cc:1939] T 43202d67804e44668c0cd338f6569b7e P 8628b8a778934989b74758dc4cabf5d7: Deleting consensus metadata
I20260620 08:57:14.604450 14680 catalog_manager.cc:4841] TS 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473): tablet 43202d67804e44668c0cd338f6569b7e (table test-table [id=13a5bf451df2475998a9713ed69c9179]) successfully deleted
W20260620 08:57:14.679132 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:14.679415 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:14.679476 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:14.683540 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:14.683629 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:14.683712 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:14.688946 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46005
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:14.689946 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:14.690919 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:14.700271 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.700273 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.700273 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:14.700474 14887 server_base.cc:1034] running on GCE node
I20260620 08:57:14.700946 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:14.701624 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:14.702878 14887 hybrid_clock.cc:648] HybridClock initialized: now 1781945834702859 us; error 26 us; skew 500 ppm
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:14.705826 14887 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:14.706873 14887 webserver.cc:460] Webserver started at http://127.0.98.2:46005/ using document root <none> and password file <none>
I20260620 08:57:14.707432 14887 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:14.707515 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:14.711449 14887 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:57:14.713565 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.714857 14887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:14.714973 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "44fc5e5fe6a24f898dc6c0ae2e326845"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "5ee8dac6f4d1c24f6433bb2051de39fe"
server_key_iv: "73c25a9115804e353df6e4039bd4da57"
server_key_version: "encryptionkey@0"
I20260620 08:57:14.715503 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:14.741663 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:14.744599 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:14.744772 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:14.745349 14887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:14.746366 14914 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 08:57:14.750020 14914 ts_tablet_manager.cc:1852] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 08:57:14.750402 14914 ts_tablet_manager.cc:1905] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:14.753983 14914 ts_tablet_manager.cc:1918] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:14.754124 14914 ts_tablet_manager.cc:1939] T 43202d67804e44668c0cd338f6569b7e P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting consensus metadata
I20260620 08:57:14.754568 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 08:57:14.754632 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260620 08:57:14.754704 14887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:14.754732 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.763424 14887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38729
I20260620 08:57:14.763464 15021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38729 every 8 connection(s)
I20260620 08:57:14.764328 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:14.768165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 14887
I20260620 08:57:14.768574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42657
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41681
--webserver_port=41779
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:14.779652 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.766596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39683 (local address 127.0.98.62:41681)
0620 08:57:14.766751 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.766756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.767627 (+ 871us) server_negotiation.cc:407] Connection header received
0620 08:57:14.768724 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.768728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.768784 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.768869 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:14.770270 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.771057 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.772646 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.772878 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.772958 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.775840 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:14.775843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:14.775845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:14.775871 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:14.777255 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.777929 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.777932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.777933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.778013 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.778422 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.778426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.778427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.778631 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:14.778731 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.779365 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.779486 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:57:14.780599 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.766994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:39683)
0620 08:57:14.767461 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.767502 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.768489 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.769053 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.769062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.769500 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:14.770122 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.770136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.771170 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.771174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.772513 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.772525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.773026 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.773031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.773246 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.773876 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:14.773898 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:14.775670 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:14.777387 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.777393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.777409 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.777785 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.778099 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.778102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.778104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.778289 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.778734 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:14.778741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:14.779079 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.779275 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.779601 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260620 08:57:14.781857 15022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41681
I20260620 08:57:14.782094 15022 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:14.782575 15022 heartbeater.cc:511] Master 127.0.98.62:41681 requested a full tablet report, sending...
I20260620 08:57:14.783676 14695 ts_manager.cc:194] Registered new tserver with Master: 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729)
I20260620 08:57:14.784247 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39683
W20260620 08:57:14.879549 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:14.879827 15025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:14.879887 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:14.883970 15025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:14.884022 15025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:14.884095 15025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:14.889484 15025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41779
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:14.890533 15025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:14.891520 15025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:57:14.900949 15025 server_base.cc:1034] running on GCE node
W20260620 08:57:14.900902 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.900880 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:14.900889 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:14.901600 15025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:14.902215 15025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:14.903378 15025 hybrid_clock.cc:648] HybridClock initialized: now 1781945834903359 us; error 30 us; skew 500 ppm
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:14.906014 15025 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:14.906847 15025 webserver.cc:460] Webserver started at http://127.0.98.3:41779/ using document root <none> and password file <none>
I20260620 08:57:14.907398 15025 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:14.907480 15025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:14.911207 15025 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:14.913480 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.914868 15025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:57:14.914976 15025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "30fc4254eb3647708c286246691fe973"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "0185ecb1ea2104cf70b10d614bcfd918"
server_key_iv: "3f845d7b6f105d602ada3ef11dee2059"
server_key_version: "encryptionkey@0"
I20260620 08:57:14.915498 15025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:14.934391 15025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:14.937264 15025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:14.937430 15025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:14.937988 15025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:14.938800 15025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:14.938853 15025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.938905 15025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:14.938932 15025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:14.947431 15025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42657
I20260620 08:57:14.947460 15153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42657 every 8 connection(s)
I20260620 08:57:14.948374 15025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:57:14.955439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15025
Jun 20 08:57:14 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:14.961339 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.950049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41405 (local address 127.0.98.62:41681)
0620 08:57:14.950196 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.950200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.951160 (+ 960us) server_negotiation.cc:407] Connection header received
0620 08:57:14.951990 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.951993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.952039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.952109 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:14.953151 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.953665 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.954743 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.954966 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.955055 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.957772 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:14.957774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:14.957776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:14.957797 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:14.959136 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.959691 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.959695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.959696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.959782 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.960199 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.960201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.960202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.960371 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:14.960441 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.961059 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.961181 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:57:14.962126 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.950432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:41405)
0620 08:57:14.951028 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.951064 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.951819 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.952247 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.952253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.952575 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:14.953028 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.953039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.953774 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.953776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.954605 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.954613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.955124 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.955127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.955268 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.955969 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:14.955995 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:14.957598 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:14.959254 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.959259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.959270 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.959584 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.959913 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:14.959916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:14.959918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:14.960102 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:14.960466 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:14.960473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:14.960774 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.960925 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.961221 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":105,"threads_started":1}
I20260620 08:57:14.963243 15154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41681
I20260620 08:57:14.963480 15154 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:14.963761 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.956157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.0.98.62:41681)
0620 08:57:14.956380 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.956384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.956469 (+ 85us) server_negotiation.cc:407] Connection header received
0620 08:57:14.956593 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.956596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.956638 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.956718 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:14.957392 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.958003 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.958951 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.959290 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.959362 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.960771 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:14.960774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:14.960776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:14.960797 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:14.962172 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.962595 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.962598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.962600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.962685 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:14.962944 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:14.962948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:14.962950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:14.963114 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:14.963310 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.963479 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.963587 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260620 08:57:14.964109 15154 heartbeater.cc:511] Master 127.0.98.62:41681 requested a full tablet report, sending...
I20260620 08:57:14.965111 14693 ts_manager.cc:194] Registered new tserver with Master: 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
I20260620 08:57:14.965674 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41405
I20260620 08:57:14.977299 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:57:14.981657 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.977985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.0.98.62:41681)
0620 08:57:14.978112 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.978116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.978231 (+ 115us) server_negotiation.cc:407] Connection header received
0620 08:57:14.978410 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.978412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.978516 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.978605 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:14.978844 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.979376 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.980268 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.980501 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.980589 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.980885 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:14.981295 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:57:14.981400 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.981463 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.981523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:57:14.982254 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:57:14.985464 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:57:14.992740 14817 tablet_service.cc:1467] Processing CreateTablet for tablet a7279fdaa6de4160ab31a245de4a0527 (DEFAULT_TABLE table=test-table [id=39276fdac57846edb7a60c2cca1d7fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:14.993398 14817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7279fdaa6de4160ab31a245de4a0527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:14.996112 14775 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Bootstrap starting.
I20260620 08:57:14.996981 14775 tablet_bootstrap.cc:654] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:14.997148 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.991656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38729 (local address 127.0.0.1:55242)
0620 08:57:14.991810 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.991825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.991916 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.992828 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.992831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.992851 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.992952 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.992958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.994489 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.994493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.995645 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.995656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.996754 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.996758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.996851 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.996873 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.996922 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.996997 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":1}
I20260620 08:57:14.997233 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.992285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.0.98.3:42657)
0620 08:57:14.992587 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.992596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.992615 (+ 19us) server_negotiation.cc:407] Connection header received
0620 08:57:14.992690 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.992694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.992858 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.992994 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.993278 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.994588 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.995660 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.996446 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.996556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.996714 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.996999 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.997095 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
I20260620 08:57:14.997013 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.991879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42657 (local address 127.0.0.1:53210)
0620 08:57:14.992122 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:14.992135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:14.992220 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:14.993080 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:14.993083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:14.993098 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:14.993175 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.993180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.994714 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.994718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.995518 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:14.995527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.996568 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.996572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:14.996662 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.996706 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 08:57:14.996795 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:57:14.996869 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
I20260620 08:57:14.997849 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:14.991876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.0.98.2:38729)
0620 08:57:14.992334 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:14.992346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:14.992367 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:57:14.992442 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:14.992447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:14.992668 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:14.992825 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:14.993103 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.994361 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.995764 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:14.996628 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:14.997522 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:14.997627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:14.997666 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 08:57:14.997730 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":119,"threads_started":1}
I20260620 08:57:14.998382 14775 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: No bootstrap required, opened a new log
I20260620 08:57:14.998471 14775 ts_tablet_manager.cc:1397] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:57:14.998924 14775 raft_consensus.cc:348] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:14.999042 14775 raft_consensus.cc:374] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:14.999081 14775 raft_consensus.cc:729] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8628b8a778934989b74758dc4cabf5d7, State: Initialized, Role: FOLLOWER
I20260620 08:57:14.999197 14775 consensus_queue.cc:260] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:14.999568 14775 ts_tablet_manager.cc:1428] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:57:14.999184 14956 tablet_service.cc:1467] Processing CreateTablet for tablet a7279fdaa6de4160ab31a245de4a0527 (DEFAULT_TABLE table=test-table [id=39276fdac57846edb7a60c2cca1d7fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:15.000245 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7279fdaa6de4160ab31a245de4a0527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:15.000661 15088 tablet_service.cc:1467] Processing CreateTablet for tablet a7279fdaa6de4160ab31a245de4a0527 (DEFAULT_TABLE table=test-table [id=39276fdac57846edb7a60c2cca1d7fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:15.001585 15088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7279fdaa6de4160ab31a245de4a0527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:15.004568 14914 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Bootstrap starting.
I20260620 08:57:15.007884 14914 tablet_bootstrap.cc:654] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:15.008777 14914 log.cc:824] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:15.010015 15168 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Bootstrap starting.
I20260620 08:57:15.011158 14914 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: No bootstrap required, opened a new log
I20260620 08:57:15.011333 14914 ts_tablet_manager.cc:1397] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 08:57:15.012171 15168 tablet_bootstrap.cc:654] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:15.013680 15168 log.cc:824] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:15.014904 14914 raft_consensus.cc:348] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.015129 14914 raft_consensus.cc:374] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:15.015192 14914 raft_consensus.cc:729] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fc5e5fe6a24f898dc6c0ae2e326845, State: Initialized, Role: FOLLOWER
I20260620 08:57:15.015587 14914 consensus_queue.cc:260] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.015903 15168 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: No bootstrap required, opened a new log
I20260620 08:57:15.016111 15168 ts_tablet_manager.cc:1397] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 08:57:15.016290 15022 heartbeater.cc:503] Master 127.0.98.62:41681 was elected leader, sending a full tablet report...
I20260620 08:57:15.016569 14914 ts_tablet_manager.cc:1428] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:57:15.019733 15168 raft_consensus.cc:348] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.020035 15168 raft_consensus.cc:374] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:15.020097 15168 raft_consensus.cc:729] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fc4254eb3647708c286246691fe973, State: Initialized, Role: FOLLOWER
I20260620 08:57:15.020586 15168 consensus_queue.cc:260] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.021248 15154 heartbeater.cc:503] Master 127.0.98.62:41681 was elected leader, sending a full tablet report...
I20260620 08:57:15.021579 15168 ts_tablet_manager.cc:1428] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 08:57:15.057199 14884 tablet.cc:2369] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:57:15.199347 15155 tablet.cc:2369] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:57:15.265467 15023 tablet.cc:2369] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:15.331933 15170 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:15.332154 15170 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.333278 15170 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657), 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473)
I20260620 08:57:15.337203 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.333563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43473 (local address 127.0.98.2:33901)
0620 08:57:15.333949 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.333963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:15.334105 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:15.334381 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:15.334384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:15.334399 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:15.334522 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:15.334528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.335213 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.335216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:15.336084 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:15.336093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.336742 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.336746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:15.336872 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:15.336896 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:57:15.336966 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:57:15.337039 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":3,"spinlock_wait_cycles":41984,"thread_start_us":100,"threads_started":1}
I20260620 08:57:15.337191 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.333744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33901 (local address 127.0.98.1:43473)
0620 08:57:15.333972 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:15.333976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:15.334023 (+ 47us) server_negotiation.cc:407] Connection header received
0620 08:57:15.334207 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:15.334210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:15.334263 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:15.334357 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:15.334621 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.335117 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.336193 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.336630 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.336726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:15.336877 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:15.336987 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:57:15.337058 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260620 08:57:15.337209 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.333680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48131 (local address 127.0.98.3:42657)
0620 08:57:15.333869 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:15.333873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:15.334059 (+ 186us) server_negotiation.cc:407] Connection header received
0620 08:57:15.334215 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:15.334218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:15.334260 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:15.334327 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:15.334581 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.335065 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.336237 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.336645 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.336734 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:15.336797 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:15.337008 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 08:57:15.337085 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 08:57:15.337229 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.333580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42657 (local address 127.0.98.2:48131)
0620 08:57:15.333971 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.333982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:15.334108 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:15.334349 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:15.334351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:15.334367 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:15.334486 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:15.334495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.335163 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.335167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:15.336136 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:15.336147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.336748 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.336752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:15.336901 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:15.336918 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 08:57:15.337026 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:57:15.337094 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":64,"spinlock_wait_cycles":5632,"thread_start_us":80,"threads_started":1}
I20260620 08:57:15.337911 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7279fdaa6de4160ab31a245de4a0527" candidate_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fc4254eb3647708c286246691fe973" is_pre_election: true
I20260620 08:57:15.338362 15108 raft_consensus.cc:2454] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fc5e5fe6a24f898dc6c0ae2e326845 in term 0.
I20260620 08:57:15.338784 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7279fdaa6de4160ab31a245de4a0527" candidate_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8628b8a778934989b74758dc4cabf5d7" is_pre_election: true
I20260620 08:57:15.338871 14910 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845; no voters:
I20260620 08:57:15.339036 14836 raft_consensus.cc:2454] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fc5e5fe6a24f898dc6c0ae2e326845 in term 0.
I20260620 08:57:15.339164 15170 raft_consensus.cc:2781] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:57:15.339263 15170 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:57:15.339318 15170 raft_consensus.cc:3037] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:15.340396 15170 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.340728 15170 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [CANDIDATE]: Term 1 election: Requested vote from peers 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657), 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473)
I20260620 08:57:15.341112 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7279fdaa6de4160ab31a245de4a0527" candidate_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8628b8a778934989b74758dc4cabf5d7"
I20260620 08:57:15.341112 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7279fdaa6de4160ab31a245de4a0527" candidate_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fc4254eb3647708c286246691fe973"
I20260620 08:57:15.341243 14836 raft_consensus.cc:3037] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:15.341243 15108 raft_consensus.cc:3037] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:15.342435 15108 raft_consensus.cc:2454] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fc5e5fe6a24f898dc6c0ae2e326845 in term 1.
I20260620 08:57:15.342435 14836 raft_consensus.cc:2454] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fc5e5fe6a24f898dc6c0ae2e326845 in term 1.
I20260620 08:57:15.342801 14910 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845; no voters:
I20260620 08:57:15.343009 15170 raft_consensus.cc:2781] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:15.343262 15170 raft_consensus.cc:686] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 LEADER]: Becoming Leader. State: Replica: 44fc5e5fe6a24f898dc6c0ae2e326845, State: Running, Role: LEADER
I20260620 08:57:15.343626 15170 consensus_queue.cc:237] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.346788 14695 catalog_manager.cc:5495] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } health_report { overall_health: UNKNOWN } } }
I20260620 08:57:15.368757 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.364611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.0.98.2:38729)
0620 08:57:15.364732 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:15.364736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:15.364749 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:57:15.364819 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:15.364821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:15.364873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:15.364962 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:15.365214 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.365803 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.367017 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:15.367303 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:15.367429 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:15.367904 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:15.368383 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:57:15.368497 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:15.368555 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 08:57:15.368619 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:57:15.374552 15108 raft_consensus.cc:1261] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Refusing update from remote peer 44fc5e5fe6a24f898dc6c0ae2e326845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:15.374550 14836 raft_consensus.cc:1261] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Refusing update from remote peer 44fc5e5fe6a24f898dc6c0ae2e326845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:15.375311 15175 consensus_queue.cc:1035] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:15.375532 15170 consensus_queue.cc:1035] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:15.381760 15178 mvcc.cc:205] Tried to move back new op lower bound from 7298850141686439936 to 7298850141570342912. Current Snapshot: MvccSnapshot[applied={T|T < 7298850141686439936}]
I20260620 08:57:15.390416 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:15.390580 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:15.392827 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Sending DeleteTablet for 3 replicas of tablet a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:15.393527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14733
I20260620 08:57:15.393493 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet a7279fdaa6de4160ab31a245de4a0527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55242
I20260620 08:57:15.393545 15088 tablet_service.cc:1514] Processing DeleteTablet for tablet a7279fdaa6de4160ab31a245de4a0527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53210
I20260620 08:57:15.394969 15188 tablet_replica.cc:330] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: stopping tablet replica
I20260620 08:57:15.395287 15188 raft_consensus.cc:2229] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:57:15.395725 15188 raft_consensus.cc:2258] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:15.397208 15188 ts_tablet_manager.cc:1905] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:57:15.399446 14680 connection.cc:535] client connection to 127.0.98.1:43473 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:57:15.399531 15189 tablet_replica.cc:330] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: stopping tablet replica
W20260620 08:57:15.399572 14680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:57:15.399855 15189 raft_consensus.cc:2229] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:15.400158 15189 raft_consensus.cc:2258] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:15.400249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14887
I20260620 08:57:15.400278 15188 ts_tablet_manager.cc:1918] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:15.402067 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.401765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43473 (local address 127.0.0.1:33338)
0620 08:57:15.401918 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.401983 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 08:57:15.402264 14680 catalog_manager.cc:4584] TS 8628b8a778934989b74758dc4cabf5d7 (127.0.98.1:43473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7279fdaa6de4160ab31a245de4a0527: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43473: connect: Connection refused (error 111)
I20260620 08:57:15.402848 15189 ts_tablet_manager.cc:1905] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:15.404942 15189 ts_tablet_manager.cc:1918] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:15.405027 15189 log.cc:1196] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:15.405303 15189 ts_tablet_manager.cc:1939] T a7279fdaa6de4160ab31a245de4a0527 P 30fc4254eb3647708c286246691fe973: Deleting consensus metadata
I20260620 08:57:15.405814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15025
I20260620 08:57:15.405995 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.405718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38729 (local address 127.0.0.1:55252)
0620 08:57:15.405868 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.405921 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 08:57:15.410552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 14662
I20260620 08:57:15.415794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41681
--webserver_interface=127.0.98.62
--webserver_port=37059
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:15.522563 15191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:15.522838 15191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:15.522894 15191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:15.526877 15191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:15.526942 15191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:15.526968 15191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:15.526988 15191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:15.527009 15191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:15.532375 15191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:15.533377 15191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:15.534353 15191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:15.542614 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:15.542625 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:15.542663 15191 server_base.cc:1034] running on GCE node
W20260620 08:57:15.542629 15199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:15.543354 15191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:15.544494 15191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:15.545663 15191 hybrid_clock.cc:648] HybridClock initialized: now 1781945835545648 us; error 25 us; skew 500 ppm
Jun 20 08:57:15 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:15.548316 15191 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:15.549162 15191 webserver.cc:460] Webserver started at http://127.0.98.62:37059/ using document root <none> and password file <none>
I20260620 08:57:15.549675 15191 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:15.549739 15191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:15.553406 15191 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:15.555363 15206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:15.556501 15191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:15.556607 15191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "bd7b0271783d4ba0bbed93fb140f678b"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "1ad8bf28bbb75d7a5aba85ef072c76b3"
server_key_iv: "73558f2dccc0101c1c39921375779016"
server_key_version: "encryptionkey@0"
I20260620 08:57:15.557106 15191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:15.571449 15191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:15.574396 15191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:15.574568 15191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:15.581434 15191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41681
I20260620 08:57:15.581743 15258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41681 every 8 connection(s)
I20260620 08:57:15.582513 15191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:15.587285 15259 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:57:15.588968 15259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap starting.
I20260620 08:57:15.591444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15191
I20260620 08:57:15.591840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41681
--webserver_port=44521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:15.607494 15259 log.cc:824] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:15.618492 15259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:15.618818 15259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b: Bootstrap complete.
I20260620 08:57:15.622551 15259 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:15.622925 15259 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7b0271783d4ba0bbed93fb140f678b, State: Initialized, Role: FOLLOWER
I20260620 08:57:15.623370 15259 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:15.623485 15259 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:15.623549 15259 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:15.623664 15259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 2 FOLLOWER]: Advancing to term 3
I20260620 08:57:15.625463 15259 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:15.625756 15259 leader_election.cc:302] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7b0271783d4ba0bbed93fb140f678b; no voters:
I20260620 08:57:15.626044 15259 leader_election.cc:288] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 08:57:15.626123 15263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 08:57:15.626298 15263 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [term 3 LEADER]: Becoming Leader. State: Replica: bd7b0271783d4ba0bbed93fb140f678b, State: Running, Role: LEADER
I20260620 08:57:15.626580 15263 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } }
I20260620 08:57:15.627225 15259 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:15.627840 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bd7b0271783d4ba0bbed93fb140f678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } } }
I20260620 08:57:15.627959 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:15.628151 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7b0271783d4ba0bbed93fb140f678b. Latest consensus state: current_term: 3 leader_uuid: "bd7b0271783d4ba0bbed93fb140f678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b0271783d4ba0bbed93fb140f678b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41681 } } }
I20260620 08:57:15.628237 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b0271783d4ba0bbed93fb140f678b [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:15.629110 15269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:15.633997 15269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:15.634630 15269 catalog_manager.cc:1259] Loaded cluster ID: eef9eaaab18b49caa22e8ef2858b8af5
I20260620 08:57:15.634689 15269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 08:57:15.635710 15269 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fda2442e9ad google::LogMessage::Fail() at ??:0
@ 0x7fda2443188a google::LogMessage::SendToLog() at ??:0
@ 0x7fda2442e4de google::LogMessage::Flush() at ??:0
@ 0x7fda2442fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda2ed6dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fda2ed6ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fda2ed6b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fda2edaf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fda266803d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda26680c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fda266823f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fda26671512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda27cb56db start_thread at ??:0
@ 0x7fda22f2c61f clone at ??:0
W20260620 08:57:15.724515 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:15.724758 15261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:15.724804 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:15.728850 15261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:15.728915 15261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:15.728978 15261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:15.734333 15261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44521
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:15.735394 15261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:15.736434 15261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:15.745867 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:15.745882 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:15.745962 15261 server_base.cc:1034] running on GCE node
W20260620 08:57:15.745863 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:15.746559 15261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:15.747323 15261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:15.748512 15261 hybrid_clock.cc:648] HybridClock initialized: now 1781945835748483 us; error 34 us; skew 500 ppm
Jun 20 08:57:15 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:15.751824 15261 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:15.752902 15261 webserver.cc:460] Webserver started at http://127.0.98.1:44521/ using document root <none> and password file <none>
I20260620 08:57:15.753609 15261 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:15.753702 15261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:15.758607 15261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:57:15.761251 15310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:15.762763 15261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 08:57:15.762884 15261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "8628b8a778934989b74758dc4cabf5d7"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "172b177d103a224215bd0a74660751fc"
server_key_iv: "c5ffe720e0424dc02a41725054c66e83"
server_key_version: "encryptionkey@0"
I20260620 08:57:15.763643 15261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:15.779822 15261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:15.783557 15261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:15.783774 15261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:15.784507 15261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:15.785652 15317 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:57:15.789767 15261 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:57:15.789845 15261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 08:57:15.789911 15261 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:57:15.791904 15261 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:57:15.791960 15261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:15.792237 15317 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Bootstrap starting.
I20260620 08:57:15.802146 15261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43473
I20260620 08:57:15.802500 15424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43473 every 8 connection(s)
I20260620 08:57:15.803364 15261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:57:15.808488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15261
I20260620 08:57:15.808979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38729
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41681
--webserver_port=46005
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:15.811215 15317 log.cc:824] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:15.819885 15317 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:15.820307 15317 tablet_bootstrap.cc:492] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Bootstrap complete.
I20260620 08:57:15.821777 15317 ts_tablet_manager.cc:1397] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.019s
I20260620 08:57:15.826100 15317 raft_consensus.cc:348] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.826711 15317 raft_consensus.cc:729] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8628b8a778934989b74758dc4cabf5d7, State: Initialized, Role: FOLLOWER
I20260620 08:57:15.827494 15317 consensus_queue.cc:260] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:15.828967 15317 ts_tablet_manager.cc:1428] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260620 08:57:15.860258 15427 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:57:15.805673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:51821)
0620 08:57:15.806276 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.806351 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:15.807527 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:15.859228 (+ 51701us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":150,"threads_started":1}
W20260620 08:57:15.861008 15425 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41681 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41681: Client connection negotiation failed: client connection to 127.0.98.62:41681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:57:15.861886 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.861599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:53725)
0620 08:57:15.861741 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.861817 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 08:57:15.862666 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:15.862437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:49573)
0620 08:57:15.862564 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:15.862615 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 08:57:15.862854 15425 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:57:15.924263 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:15.924532 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:15.924578 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:15.928638 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:15.928699 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:15.928771 15428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:15.934078 15428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46005
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:15.935078 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:15.936085 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:15.947276 15436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:15.947273 15438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:15.947381 15435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:15.947926 15428 server_base.cc:1034] running on GCE node
I20260620 08:57:15.948333 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:15.948966 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:15.950132 15428 hybrid_clock.cc:648] HybridClock initialized: now 1781945835950110 us; error 31 us; skew 500 ppm
Jun 20 08:57:15 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:15.952705 15428 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:15.953572 15428 webserver.cc:460] Webserver started at http://127.0.98.2:46005/ using document root <none> and password file <none>
I20260620 08:57:15.954092 15428 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:15.954154 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:15.957828 15428 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:15.959816 15445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:15.960908 15428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:15.961112 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "44fc5e5fe6a24f898dc6c0ae2e326845"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "5ee8dac6f4d1c24f6433bb2051de39fe"
server_key_iv: "73c25a9115804e353df6e4039bd4da57"
server_key_version: "encryptionkey@0"
I20260620 08:57:15.961637 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:15.979044 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:15.982597 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:15.982784 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:15.983377 15428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:15.984521 15452 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 08:57:15.988168 15452 ts_tablet_manager.cc:1852] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 08:57:15.988564 15452 ts_tablet_manager.cc:1905] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:15.992251 15452 ts_tablet_manager.cc:1918] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:15.992355 15452 log.cc:1196] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:15.992597 15452 ts_tablet_manager.cc:1939] T a7279fdaa6de4160ab31a245de4a0527 P 44fc5e5fe6a24f898dc6c0ae2e326845: Deleting consensus metadata
I20260620 08:57:15.994961 15428 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 08:57:15.995033 15428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.000s
I20260620 08:57:15.995105 15428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:15.995131 15428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:16.003726 15428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38729
I20260620 08:57:16.003795 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38729 every 8 connection(s)
I20260620 08:57:16.004624 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:16.006140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15428
I20260620 08:57:16.006484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42657
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41681
--webserver_port=41779
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:16.009153 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.006892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:39097)
0620 08:57:16.008093 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.008233 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":109,"threads_started":1}
W20260620 08:57:16.009577 15560 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41681 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41681: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
I20260620 08:57:16.010178 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.009944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:35407)
0620 08:57:16.010061 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.010114 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:57:16.010839 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.010612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:54651)
0620 08:57:16.010718 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.010767 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 08:57:16.010999 15560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:57:16.112389 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:16.112639 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:16.112681 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:16.116631 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:16.116688 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:16.116746 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:16.121796 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41779
--tserver_master_addrs=127.0.98.62:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:16.122728 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:16.123662 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:16.133564 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:16.133548 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:16.133648 15563 server_base.cc:1034] running on GCE node
W20260620 08:57:16.133548 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:16.134145 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:16.134758 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:16.135946 15563 hybrid_clock.cc:648] HybridClock initialized: now 1781945836135919 us; error 32 us; skew 500 ppm
Jun 20 08:57:16 dist-test-slave-fb1v krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:16.138526 15563 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:16.139343 15563 webserver.cc:460] Webserver started at http://127.0.98.3:41779/ using document root <none> and password file <none>
I20260620 08:57:16.139895 15563 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:16.139961 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:16.143532 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:16.145418 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:16.146535 15563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:16.146626 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "30fc4254eb3647708c286246691fe973"
format_stamp: "Formatted at 2026-06-20 08:57:13 on dist-test-slave-fb1v"
server_key: "0185ecb1ea2104cf70b10d614bcfd918"
server_key_iv: "3f845d7b6f105d602ada3ef11dee2059"
server_key_version: "encryptionkey@0"
I20260620 08:57:16.147148 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:16.163030 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:16.166407 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:16.166564 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:16.167088 15563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:16.167925 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:16.167974 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:16.168027 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:16.168056 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:16.176636 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42657
I20260620 08:57:16.176812 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42657 every 8 connection(s)
I20260620 08:57:16.177613 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:57:16.181205 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.179780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:35597)
0620 08:57:16.180445 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.180546 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":120,"threads_started":1}
W20260620 08:57:16.181627 15692 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41681 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41681: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
I20260620 08:57:16.182261 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.182000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:51479)
0620 08:57:16.182136 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.182203 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 08:57:16.182837 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.182647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:39667)
0620 08:57:16.182740 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.182785 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:57:16.183065 15692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:57:16.183429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15563
I20260620 08:57:16.863942 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:16.863515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:37601)
0620 08:57:16.863767 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:16.863837 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
I20260620 08:57:17.012209 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.011636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:36021)
0620 08:57:17.011900 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:17.011959 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260620 08:57:17.184031 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.183600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:41889)
0620 08:57:17.183878 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:17.183931 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260620 08:57:17.606025 15699 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:17.606207 15699 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:17.607407 15699 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729), 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
Jun 20 08:57:17 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 08:57:17 dist-test-slave-fb1v krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 08:57:17.621044 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.607699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42657 (local address 127.0.98.1:59201)
0620 08:57:17.608227 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:17.608242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:17.608358 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:17.610148 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:17.610154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:17.610688 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:17.611153 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:17.611169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.612454 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.612457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:17.613384 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:17.613393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.614042 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.614046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:17.614240 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:17.614834 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:17.614850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:17.616659 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:17.618716 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:17.618723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:17.618735 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:17.619077 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:17.619375 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:17.619378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:17.619380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:17.619512 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:17.620116 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:17.620122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:17.620345 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 08:57:17.620501 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 08:57:17.620820 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":189,"spinlock_wait_cycles":3840,"thread_start_us":102,"threads_started":1}
I20260620 08:57:17.621052 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.608069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59201 (local address 127.0.98.3:42657)
0620 08:57:17.608659 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:17.608680 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:17.608723 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:57:17.609431 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:17.609437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:17.609781 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:17.610056 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:17.611341 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.612326 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.613493 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.613932 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.614076 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:17.616820 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:17.616825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:17.616834 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:17.616861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:17.618610 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:17.619177 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:17.619183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:17.619188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:17.619291 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:17.619596 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:17.619599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:17.619601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:17.620005 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:17.620120 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:17.620634 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 08:57:17.620871 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":129,"threads_started":1}
I20260620 08:57:17.621057 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.608013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41457 (local address 127.0.98.2:38729)
0620 08:57:17.608548 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:17.608565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:17.608619 (+ 54us) server_negotiation.cc:407] Connection header received
0620 08:57:17.609744 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:17.609754 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:17.610066 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:17.610349 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:17.611324 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.612323 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.613495 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.613933 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.614076 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:17.616555 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:17.616562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:17.616577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:17.616611 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:17.618645 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:17.619186 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:17.619190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:17.619194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:17.619299 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:17.619630 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:17.619634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:17.619636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:17.620006 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:17.620165 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:17.620624 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 08:57:17.620875 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":83,"threads_started":1}
I20260620 08:57:17.621069 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.607831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38729 (local address 127.0.98.1:41457)
0620 08:57:17.608363 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:17.608375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:17.608436 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:17.610330 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:17.610334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:17.610708 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:17.611153 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:17.611169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.612460 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.612463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:17.613388 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:17.613397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:17.614042 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:17.614045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:17.614244 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:17.614893 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:17.614912 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:17.616335 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:17.618751 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:17.618753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:17.618754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:17.619086 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:17.619387 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:17.619391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:17.619393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:17.619518 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:17.620134 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:17.620137 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:17.620344 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 08:57:17.620501 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 08:57:17.620898 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":309,"spinlock_wait_cycles":28928,"thread_start_us":123,"threads_started":1}
W20260620 08:57:17.623667 15314 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
W20260620 08:57:17.624258 15313 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:17.624326 15313 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8628b8a778934989b74758dc4cabf5d7; no voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845
I20260620 08:57:17.624536 15699 raft_consensus.cc:2726] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:57:17.865028 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:17.864750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:47371)
0620 08:57:17.864875 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:17.864940 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:57:18.013947 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:18.013352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:56967)
0620 08:57:18.013724 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:18.013797 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":131,"threads_started":1}
I20260620 08:57:18.185216 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:18.184788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:56075)
0620 08:57:18.185058 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:18.185124 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260620 08:57:18.866274 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:18.865820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:57033)
0620 08:57:18.866112 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:18.866178 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260620 08:57:19.015440 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:19.014968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:54483)
0620 08:57:19.015260 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:19.015322 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260620 08:57:19.169240 15708 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:19.169375 15708 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:19.169781 15708 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729), 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
W20260620 08:57:19.170503 15314 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
W20260620 08:57:19.170607 15313 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:19.170692 15313 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8628b8a778934989b74758dc4cabf5d7; no voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845
I20260620 08:57:19.170818 15708 raft_consensus.cc:2726] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:57:19.186309 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:19.185927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:47791)
0620 08:57:19.186157 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:19.186207 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260620 08:57:19.867437 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:19.867009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:58725)
0620 08:57:19.867274 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:19.867335 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260620 08:57:20.017218 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:20.016590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:46033)
0620 08:57:20.016985 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:20.017077 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":105,"threads_started":1}
I20260620 08:57:20.187423 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:20.187037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:55857)
0620 08:57:20.187273 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:20.187334 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260620 08:57:20.868975 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:20.868508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:51373)
0620 08:57:20.868816 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:20.868873 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":77,"threads_started":1}
I20260620 08:57:20.922860 15714 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:20.922992 15714 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:20.923415 15714 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729), 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
W20260620 08:57:20.924155 15314 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
W20260620 08:57:20.924253 15313 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:20.924345 15313 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8628b8a778934989b74758dc4cabf5d7; no voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845
I20260620 08:57:20.924491 15714 raft_consensus.cc:2726] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:57:21.018949 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:21.018475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:54399)
0620 08:57:21.018783 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:21.018849 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260620 08:57:21.188441 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:21.188075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:42579)
0620 08:57:21.188307 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:21.188355 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260620 08:57:21.870141 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:21.869719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:34925)
0620 08:57:21.869976 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:21.870045 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260620 08:57:22.020507 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:22.019859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:47331)
0620 08:57:22.020264 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:22.020361 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":101,"threads_started":1}
I20260620 08:57:22.189569 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:22.189124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:54183)
0620 08:57:22.189402 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:22.189470 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260620 08:57:22.871234 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:22.870828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:50159)
0620 08:57:22.871075 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:22.871135 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260620 08:57:23.021863 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:23.021374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:58137)
0620 08:57:23.021691 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:23.021754 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260620 08:57:23.191185 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:23.190634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:47051)
0620 08:57:23.191004 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:23.191066 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
I20260620 08:57:23.872330 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:23.871939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:37413)
0620 08:57:23.872168 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:23.872233 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260620 08:57:24.023175 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:24.022677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:55481)
0620 08:57:24.022985 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:24.023054 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260620 08:57:24.192451 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:24.192031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:48275)
0620 08:57:24.192283 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:24.192340 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20260620 08:57:24.873570 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:24.873108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:44659)
0620 08:57:24.873402 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:24.873466 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260620 08:57:25.024503 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:25.024056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:38263)
0620 08:57:25.024345 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:25.024407 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260620 08:57:25.125842 15728 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:25.126001 15728 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:25.126518 15728 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729), 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
W20260620 08:57:25.127260 15313 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
W20260620 08:57:25.127357 15314 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:25.127449 15314 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8628b8a778934989b74758dc4cabf5d7; no voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845
I20260620 08:57:25.127591 15728 raft_consensus.cc:2726] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:57:25.193789 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:25.193327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:37023)
0620 08:57:25.193633 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:25.193696 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
I20260620 08:57:25.874742 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:25.874355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:41859)
0620 08:57:25.874595 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:25.874649 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260620 08:57:26.025785 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:26.025361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:52705)
0620 08:57:26.025619 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:26.025679 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260620 08:57:26.195191 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:26.194772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:39809)
0620 08:57:26.195037 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:26.195096 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260620 08:57:26.876014 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:26.875562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:49203)
0620 08:57:26.875859 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:26.875918 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":122,"threads_started":1}
I20260620 08:57:27.026974 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:27.026548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:36507)
0620 08:57:27.026817 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:27.026879 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260620 08:57:27.196367 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:27.195967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:48255)
0620 08:57:27.196214 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:27.196268 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260620 08:57:27.877043 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:27.876650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:37123)
0620 08:57:27.876897 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:27.876950 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260620 08:57:28.028405 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:28.027868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:33713)
0620 08:57:28.028211 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:28.028288 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260620 08:57:28.197659 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:28.197209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:33533)
0620 08:57:28.197494 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:28.197554 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260620 08:57:28.856406 15739 raft_consensus.cc:482] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:28.856529 15739 raft_consensus.cc:504] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fc5e5fe6a24f898dc6c0ae2e326845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38729 } } peers { permanent_uuid: "30fc4254eb3647708c286246691fe973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42657 } } peers { permanent_uuid: "8628b8a778934989b74758dc4cabf5d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43473 } }
I20260620 08:57:28.856904 15739 leader_election.cc:288] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729), 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657)
W20260620 08:57:28.857582 15313 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30fc4254eb3647708c286246691fe973 (127.0.98.3:42657): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
W20260620 08:57:28.857681 15314 leader_election.cc:341] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fc5e5fe6a24f898dc6c0ae2e326845 (127.0.98.2:38729): Not found: Tablet not found: a7279fdaa6de4160ab31a245de4a0527
I20260620 08:57:28.857767 15314 leader_election.cc:302] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8628b8a778934989b74758dc4cabf5d7; no voters: 30fc4254eb3647708c286246691fe973, 44fc5e5fe6a24f898dc6c0ae2e326845
I20260620 08:57:28.857903 15739 raft_consensus.cc:2726] T a7279fdaa6de4160ab31a245de4a0527 P 8628b8a778934989b74758dc4cabf5d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:57:28.878139 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:28.877717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:49525)
0620 08:57:28.877987 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:28.878045 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260620 08:57:29.029706 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:29.029230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:56733)
0620 08:57:29.029529 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:29.029595 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260620 08:57:29.198719 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:29.198318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:57557)
0620 08:57:29.198573 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:29.198626 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260620 08:57:29.879423 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:29.878934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:50725)
0620 08:57:29.879239 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:29.879325 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260620 08:57:30.031033 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:30.030525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:57615)
0620 08:57:30.030863 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:30.030937 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
I20260620 08:57:30.200085 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:30.199692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:53783)
0620 08:57:30.199923 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:30.199989 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260620 08:57:30.880681 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:30.880303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.1:59347)
0620 08:57:30.880538 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:30.880589 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260620 08:57:31.032291 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:31.031898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.2:53587)
0620 08:57:31.032143 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:31.032196 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260620 08:57:31.194237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15261
I20260620 08:57:31.199657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15428
I20260620 08:57:31.201332 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:31.200932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41681 (local address 127.0.98.3:42393)
0620 08:57:31.201179 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:31.201237 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260620 08:57:31.204964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15563
2026-06-20T08:57:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19187 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:57:31 dist-test-slave-fb1v krb5kdc[15751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:57:31 dist-test-slave-fb1v krb5kdc[15751](info): set up 2 sockets
Jun 20 08:57:31 dist-test-slave-fb1v krb5kdc[15751](info): commencing operation
krb5kdc: starting...
W20260620 08:57:32.167501 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:57:32.280102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:32.385306 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:32.385600 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:32.385667 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:32.389825 15764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:57:32.389884 15764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:57:32.389906 15764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:57:32.389925 15764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:32.389950 15764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:32.389982 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:32.390010 15764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:32.390038 15764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:32.395305 15764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:32.396381 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:32.397353 15764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:32.405346 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:32.405496 15764 server_base.cc:1034] running on GCE node
W20260620 08:57:32.405376 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:32.405360 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:32.406159 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:32.407178 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:32.408339 15764 hybrid_clock.cc:648] HybridClock initialized: now 1781945852408323 us; error 30 us; skew 500 ppm
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:32.415508 15764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:32.416456 15764 webserver.cc:460] Webserver started at http://127.0.98.62:40811/ using document root <none> and password file <none>
I20260620 08:57:32.416992 15764 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:32.417053 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:32.417239 15764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:32.419203 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "87d5c93b3ddf4572bb505bdadb7d73cb"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "5985c8ee0954319afd21f03ed0f6b186"
server_key_iv: "4aa7b699e5c34144c12752f862f9b1fb"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.419754 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "87d5c93b3ddf4572bb505bdadb7d73cb"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "5985c8ee0954319afd21f03ed0f6b186"
server_key_iv: "4aa7b699e5c34144c12752f862f9b1fb"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.423413 15764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:57:32.425887 15779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.427138 15764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:32.427253 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "87d5c93b3ddf4572bb505bdadb7d73cb"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "5985c8ee0954319afd21f03ed0f6b186"
server_key_iv: "4aa7b699e5c34144c12752f862f9b1fb"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.427398 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:32.445994 15764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:32.449002 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:32.449151 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:32.455472 15764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45763
I20260620 08:57:32.455780 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45763 every 8 connection(s)
I20260620 08:57:32.456430 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:32.459085 15832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:32.464605 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap starting.
I20260620 08:57:32.465998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15764
I20260620 08:57:32.466106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:57:32.466382 392 external_mini_cluster.cc:1428] Setting key 73afe2c4237e1bb0d70bda14fadc9bac
I20260620 08:57:32.467037 15832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:32.467882 15832 log.cc:824] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:32.469857 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: No bootstrap required, opened a new log
I20260620 08:57:32.473851 15832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:32.474189 15832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:32.474249 15832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d5c93b3ddf4572bb505bdadb7d73cb, State: Initialized, Role: FOLLOWER
I20260620 08:57:32.474805 15832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:32.474979 15832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:32.475044 15832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:32.475136 15832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:32.476148 15832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:32.476500 15832 leader_election.cc:302] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d5c93b3ddf4572bb505bdadb7d73cb; no voters:
I20260620 08:57:32.476848 15832 leader_election.cc:288] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:57:32.476951 15837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:32.477190 15837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 LEADER]: Becoming Leader. State: Replica: 87d5c93b3ddf4572bb505bdadb7d73cb, State: Running, Role: LEADER
I20260620 08:57:32.477517 15837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:32.478176 15832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:32.479115 15838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260620 08:57:32.479238 15838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:32.479561 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d5c93b3ddf4572bb505bdadb7d73cb. Latest consensus state: current_term: 1 leader_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260620 08:57:32.479746 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:32.481693 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:32.467394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.98.62:45763)
0620 08:57:32.467794 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:32.467810 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:32.467844 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:57:32.468470 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:32.468486 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:32.468799 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:32.469124 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:32.469950 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.470986 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.472275 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.472976 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.473254 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:32.476017 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:32.476024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:32.476041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:32.476078 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:32.477993 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:32.478603 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:32.478609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:32.478614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:32.478746 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:32.479059 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:32.479063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:32.479065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:32.479629 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:32.479824 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:32.480703 (+ 879us) server_negotiation.cc:299] Negotiation successful
0620 08:57:32.480991 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
W20260620 08:57:32.483750 15852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:57:32.483814 15852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:57:32.483865 15846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:32.484537 15846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:32.489547 15846 catalog_manager.cc:1347] Generated new cluster ID: b4b0ba4410a649f3ba2bb2e916d6dfa1
I20260620 08:57:32.489600 15846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:32.527966 15846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:57:32.529099 15846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:32.538708 15846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Generated new TSK 0
I20260620 08:57:32.539413 15846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:57:32.570186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:32.695572 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:32.695844 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:32.695885 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:32.699785 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:32.699837 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:32.699892 15858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:32.704967 15858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:32.705986 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:32.706930 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:32.715925 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:32.716014 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:32.716006 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:32.716178 15858 server_base.cc:1034] running on GCE node
I20260620 08:57:32.716588 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:32.717170 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:32.718348 15858 hybrid_clock.cc:648] HybridClock initialized: now 1781945852718331 us; error 34 us; skew 500 ppm
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:32.721055 15858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:32.721935 15858 webserver.cc:460] Webserver started at http://127.0.98.1:35421/ using document root <none> and password file <none>
I20260620 08:57:32.722469 15858 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:32.722534 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:32.722738 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:32.724774 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "27dcf0338b164a9aa916defcd8317765"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "1ba84319372bcd2d7ddb7065212c31c8"
server_key_iv: "0ed9563cd683e51634de604f4521fba5"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.725282 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "27dcf0338b164a9aa916defcd8317765"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "1ba84319372bcd2d7ddb7065212c31c8"
server_key_iv: "0ed9563cd683e51634de604f4521fba5"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.728880 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:57:32.731159 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.732394 15858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:57:32.732504 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "27dcf0338b164a9aa916defcd8317765"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "1ba84319372bcd2d7ddb7065212c31c8"
server_key_iv: "0ed9563cd683e51634de604f4521fba5"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.732600 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:32.754689 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:32.757831 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:32.758013 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:32.758606 15858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:32.759418 15858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:32.759465 15858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.759644 15858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:32.759680 15858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.768050 15858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41839
I20260620 08:57:32.768141 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41839 every 8 connection(s)
I20260620 08:57:32.768926 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:57:32.777031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15858
I20260620 08:57:32.777174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:57:32.777477 392 external_mini_cluster.cc:1428] Setting key 318269331d01e70757f15a4f0b061be2
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:32.783025 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:32.770769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33599 (local address 127.0.98.62:45763)
0620 08:57:32.770914 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:32.770918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:32.771979 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 08:57:32.772807 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:32.772810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:32.772856 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:32.772931 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:32.774121 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.774788 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.776149 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.776440 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.776533 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:32.779347 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:32.779351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:32.779354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:32.779380 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:32.781013 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:32.781497 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:32.781500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:32.781501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:32.781590 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:32.781921 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:32.781924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:32.781925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:32.782101 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:32.782175 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:32.782774 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 08:57:32.782904 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:57:32.784188 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:32.771161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:33599)
0620 08:57:32.771839 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:32.771879 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:32.772610 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:32.773064 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:32.773071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:32.773403 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:32.773847 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:32.773859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.774966 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.774969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:32.776000 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:32.776011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.776600 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.776603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:32.776764 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:32.777459 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:32.777475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:32.779132 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:32.781130 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:32.781135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:32.781147 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:32.781385 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:32.781665 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:32.781668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:32.781670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:32.781833 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:32.782187 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:32.782192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:32.782488 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 08:57:32.782697 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 08:57:32.783058 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":122,"threads_started":1}
I20260620 08:57:32.785383 15987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260620 08:57:32.785620 15987 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:32.786098 15987 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:57:32.787531 15796 ts_manager.cc:194] Registered new tserver with Master: 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839)
I20260620 08:57:32.788743 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:57:32.804697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:32.908980 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:32.909258 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:32.909303 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:32.913375 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:32.913437 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:32.913496 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:32.919384 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:32.920961 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:32.922770 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:32.933542 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:32.933617 15992 server_base.cc:1034] running on GCE node
W20260620 08:57:32.933547 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:32.933548 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:32.934231 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:32.934877 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:32.936060 15992 hybrid_clock.cc:648] HybridClock initialized: now 1781945852936037 us; error 39 us; skew 500 ppm
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:32.938660 15992 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:32.939464 15992 webserver.cc:460] Webserver started at http://127.0.98.2:38031/ using document root <none> and password file <none>
I20260620 08:57:32.940001 15992 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:32.940070 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:32.940227 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:32.942111 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "aeb1a1bb184a302b0b6a166fb4a12145"
server_key_iv: "f6e37aefc57e7d3d85b838698393ab97"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.942608 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "aeb1a1bb184a302b0b6a166fb4a12145"
server_key_iv: "f6e37aefc57e7d3d85b838698393ab97"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.946126 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 08:57:32.948436 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.949606 15992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:32.949710 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "aeb1a1bb184a302b0b6a166fb4a12145"
server_key_iv: "f6e37aefc57e7d3d85b838698393ab97"
server_key_version: "encryptionkey@0"
I20260620 08:57:32.949813 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:32.972561 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:32.975669 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:32.975833 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:32.976397 15992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:32.977221 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:32.977268 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.977320 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:32.977345 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:32.985827 15992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34817
I20260620 08:57:32.985845 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34817 every 8 connection(s)
I20260620 08:57:32.986709 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:32.990532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 15992
I20260620 08:57:32.990651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:57:32.990964 392 external_mini_cluster.cc:1428] Setting key 849b8b9132601a0121403c459e8b0b6f
Jun 20 08:57:32 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:57:33.001241 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:32.988509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34333 (local address 127.0.98.62:45763)
0620 08:57:32.988646 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:32.988650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:32.989748 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 08:57:32.990609 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:32.990613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:32.990669 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:32.990756 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:32.991983 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.992603 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.994177 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.994510 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.994620 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:32.997318 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:32.997322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:32.997326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:32.997354 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:32.998969 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:32.999668 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:32.999672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:32.999674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:32.999762 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:33.000139 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:33.000142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:33.000144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:33.000415 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:33.000521 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.001011 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.001113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:57:33.002091 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:32.988863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:34333)
0620 08:57:32.989588 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:32.989628 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:32.990409 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:32.990893 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:32.990899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:32.991285 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:32.991858 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:32.991872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.992739 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.992757 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:32.994041 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:32.994054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:32.994676 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:32.994679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:32.994848 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:32.995451 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:32.995470 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:32.997144 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:32.999250 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:32.999254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:32.999267 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:32.999533 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:32.999868 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:32.999870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:32.999871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:33.000020 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:33.000540 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:33.000546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:33.000791 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.000962 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.001250 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:57:33.003279 16121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260620 08:57:33.003530 16121 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:33.004119 16121 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260620 08:57:33.005306 15796 ts_manager.cc:194] Registered new tserver with Master: 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
I20260620 08:57:33.005897 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:57:33.020402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:33.127081 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:33.127341 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:33.127400 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:33.131523 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:33.131578 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:33.131675 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:33.137151 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:33.138185 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:33.139160 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:57:33.148391 16126 server_base.cc:1034] running on GCE node
W20260620 08:57:33.148372 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:33.148361 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:33.148591 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:33.148963 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:33.149577 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:33.150743 16126 hybrid_clock.cc:648] HybridClock initialized: now 1781945853150716 us; error 38 us; skew 500 ppm
Jun 20 08:57:33 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:33.153687 16126 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:33.154495 16126 webserver.cc:460] Webserver started at http://127.0.98.3:46699/ using document root <none> and password file <none>
I20260620 08:57:33.155004 16126 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:33.155066 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:33.155241 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:57:33.157236 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "b016d74c13c44108be60bc9829b29a09"
format_stamp: "Formatted at 2026-06-20 08:57:33 on dist-test-slave-fb1v"
server_key: "3d1f9e022d5cb06829850867f0745e73"
server_key_iv: "61c610aa7faa9d2c06c273e12c9f2974"
server_key_version: "encryptionkey@0"
I20260620 08:57:33.157743 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b016d74c13c44108be60bc9829b29a09"
format_stamp: "Formatted at 2026-06-20 08:57:33 on dist-test-slave-fb1v"
server_key: "3d1f9e022d5cb06829850867f0745e73"
server_key_iv: "61c610aa7faa9d2c06c273e12c9f2974"
server_key_version: "encryptionkey@0"
I20260620 08:57:33.161312 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 08:57:33.163817 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:33.165109 16126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:33.165220 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "b016d74c13c44108be60bc9829b29a09"
format_stamp: "Formatted at 2026-06-20 08:57:33 on dist-test-slave-fb1v"
server_key: "3d1f9e022d5cb06829850867f0745e73"
server_key_iv: "61c610aa7faa9d2c06c273e12c9f2974"
server_key_version: "encryptionkey@0"
I20260620 08:57:33.165324 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:33.194343 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:33.198087 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:33.198287 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:33.198894 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:33.199746 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:33.199796 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:33.199841 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:33.199870 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:33.208658 16126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33225
I20260620 08:57:33.208684 16254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33225 every 8 connection(s)
I20260620 08:57:33.209904 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:57:33.216657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16126
I20260620 08:57:33.216794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:57:33.217345 392 external_mini_cluster.cc:1428] Setting key 1735b42807769a4203af224dda5e7459
Jun 20 08:57:33 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:33.225668 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.212406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33927 (local address 127.0.98.62:45763)
0620 08:57:33.212567 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.212572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.213577 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 08:57:33.214445 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.214448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.214501 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.214578 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:33.215826 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.216376 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.218340 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.218617 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.218716 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.221462 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:33.221466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:33.221468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:33.221493 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:33.222813 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:33.223704 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:33.223707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:33.223708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:33.223792 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:33.224254 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:33.224257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:33.224258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:33.224465 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:33.224551 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.225417 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.225542 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:57:33.226524 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.212612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:33927)
0620 08:57:33.213398 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.213443 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:33.214233 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:33.214756 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:33.214763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:33.215142 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:33.215666 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.215685 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.216534 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.216538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.218128 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.218148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.218783 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.218789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.218976 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.219742 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:33.219759 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:33.221276 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:33.223043 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:33.223051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:33.223071 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:33.223519 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:33.223921 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:33.223924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:33.223926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:33.224119 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:33.224653 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:33.224660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:33.225037 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.225330 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.225670 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":139,"threads_started":1}
I20260620 08:57:33.227962 16255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260620 08:57:33.228195 16255 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:33.228736 16255 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260620 08:57:33.229779 15796 ts_manager.cc:194] Registered new tserver with Master: b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225)
I20260620 08:57:33.230335 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33927
I20260620 08:57:33.231638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:57:33.240661 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.233501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.98.62:45763)
0620 08:57:33.233651 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.233655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.233814 (+ 159us) server_negotiation.cc:407] Connection header received
0620 08:57:33.233925 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.233928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.233971 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.234036 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:33.234720 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.235254 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.236208 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.236433 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.236518 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.237818 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:33.237822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:33.237824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:33.237850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:33.239123 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:33.239507 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:33.239511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:33.239513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:33.239587 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:33.239833 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:33.239836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:33.239838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:33.240062 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:33.240185 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.240403 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.240511 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:57:33.243412 15796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:57:33.246383 15796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:57:33.260542 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.254572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41839 (local address 127.0.0.1:43246)
0620 08:57:33.255384 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.255466 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:33.255582 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:33.256312 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:33.256317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:33.256334 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:33.256508 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.256519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.257923 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.257926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.259071 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.259082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.259932 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.259936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.260052 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.260090 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.260196 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.260353 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260620 08:57:33.260620 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.255148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.0.98.2:34817)
0620 08:57:33.255796 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.255806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.256025 (+ 219us) server_negotiation.cc:407] Connection header received
0620 08:57:33.256133 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.256139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.256319 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.256581 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:33.256773 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.257557 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.259225 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.260051 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.260154 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.260281 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.260384 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.260476 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":89,"threads_started":1}
I20260620 08:57:33.260934 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.255371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.0.98.3:33225)
0620 08:57:33.255820 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.255834 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.256136 (+ 302us) server_negotiation.cc:407] Connection header received
0620 08:57:33.256216 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.256220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.256382 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.256587 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:33.256971 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.258164 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.259244 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.260051 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.260154 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.260279 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.260699 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.260806 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":90,"threads_started":1}
I20260620 08:57:33.261212 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.254814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.0.98.1:41839)
0620 08:57:33.255535 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.255549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.255573 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:57:33.255798 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.255804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.256011 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.256155 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:33.256652 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.257795 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.259197 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.259812 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.259910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.260062 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.260980 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.261076 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":76,"threads_started":1}
I20260620 08:57:33.260542 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.254572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34817 (local address 127.0.0.1:35188)
0620 08:57:33.255922 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.255933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:33.256027 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:33.256556 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:33.256559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:33.256578 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:33.256669 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.256674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.257684 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.257687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.259085 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.259091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.260150 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.260154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.260246 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.260265 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.260345 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.260413 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"mutex_wait_us":58,"thread_start_us":87,"threads_started":1}
I20260620 08:57:33.260778 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.254739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33225 (local address 127.0.0.1:49788)
0620 08:57:33.256020 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.256032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:33.256117 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:33.256746 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:33.256749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:33.256766 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:33.256854 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.256860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.258292 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.258295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.259142 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.259153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.260358 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.260361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.260446 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.260468 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.260535 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.260599 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":105,"threads_started":1}
I20260620 08:57:33.264072 16189 tablet_service.cc:1467] Processing CreateTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc (DEFAULT_TABLE table=test-table [id=d4e7c0aed1d74067803d09365f651358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:33.265035 16189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631c4da0ce6c45e2806f440b2911e1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:33.264842 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc (DEFAULT_TABLE table=test-table [id=d4e7c0aed1d74067803d09365f651358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:33.264842 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc (DEFAULT_TABLE table=test-table [id=d4e7c0aed1d74067803d09365f651358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:33.266201 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631c4da0ce6c45e2806f440b2911e1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:33.266201 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631c4da0ce6c45e2806f440b2911e1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:33.271675 16272 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Bootstrap starting.
I20260620 08:57:33.273124 16274 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Bootstrap starting.
I20260620 08:57:33.273554 16273 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Bootstrap starting.
I20260620 08:57:33.274085 16272 tablet_bootstrap.cc:654] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:33.275308 16272 log.cc:824] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:33.275650 16273 tablet_bootstrap.cc:654] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:33.276834 16273 log.cc:824] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:33.277909 16272 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: No bootstrap required, opened a new log
I20260620 08:57:33.278183 16272 ts_tablet_manager.cc:1397] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 08:57:33.279052 16273 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: No bootstrap required, opened a new log
I20260620 08:57:33.279289 16273 ts_tablet_manager.cc:1397] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:57:33.279786 16274 tablet_bootstrap.cc:654] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:33.280622 16274 log.cc:824] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:33.282029 16272 raft_consensus.cc:348] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.282310 16272 raft_consensus.cc:374] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:33.282397 16272 raft_consensus.cc:729] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b016d74c13c44108be60bc9829b29a09, State: Initialized, Role: FOLLOWER
I20260620 08:57:33.282323 16273 raft_consensus.cc:348] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.282558 16273 raft_consensus.cc:374] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:33.282626 16273 raft_consensus.cc:729] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dcf0338b164a9aa916defcd8317765, State: Initialized, Role: FOLLOWER
I20260620 08:57:33.282713 16274 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: No bootstrap required, opened a new log
I20260620 08:57:33.282886 16274 ts_tablet_manager.cc:1397] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20260620 08:57:33.282851 16272 consensus_queue.cc:260] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.283015 16273 consensus_queue.cc:260] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.283867 16272 ts_tablet_manager.cc:1428] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 08:57:33.284278 16255 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260620 08:57:33.284317 16273 ts_tablet_manager.cc:1428] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:57:33.284337 15987 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260620 08:57:33.286288 16274 raft_consensus.cc:348] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.286496 16274 raft_consensus.cc:374] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:33.286556 16274 raft_consensus.cc:729] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdefe194f6c4a1ba4ad6296af8ea0e9, State: Initialized, Role: FOLLOWER
I20260620 08:57:33.286912 16274 consensus_queue.cc:260] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.287678 16121 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260620 08:57:33.287837 16274 ts_tablet_manager.cc:1428] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260620 08:57:33.461228 16256 tablet.cc:2369] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:33.478896 16280 raft_consensus.cc:482] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:33.479118 16280 raft_consensus.cc:504] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.480369 16280 leader_election.cc:288] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839), b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225)
I20260620 08:57:33.484155 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.480742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40357 (local address 127.0.98.1:41839)
0620 08:57:33.480864 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.480869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.481145 (+ 276us) server_negotiation.cc:407] Connection header received
0620 08:57:33.481254 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.481257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.481304 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.481379 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:33.481624 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.482126 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.483185 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.483582 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.483713 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.483786 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.483950 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.484029 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:57:33.484162 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.480742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48303 (local address 127.0.98.3:33225)
0620 08:57:33.480864 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.480869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.480922 (+ 53us) server_negotiation.cc:407] Connection header received
0620 08:57:33.481067 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.481071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.481124 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.481198 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:33.481468 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.482012 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.483095 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.483526 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.483625 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.483696 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.483991 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.484062 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:57:33.484155 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.480652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33225 (local address 127.0.98.2:48303)
0620 08:57:33.480852 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.480868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:33.480966 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:33.481211 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:33.481214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:33.481230 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:33.481351 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.481359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.482132 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.482135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.482967 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.482976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.483699 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.483703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.483809 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.483875 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.483947 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.484029 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":40}
I20260620 08:57:33.484175 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.480652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41839 (local address 127.0.98.2:40357)
0620 08:57:33.481035 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.481048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:33.481160 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:33.481394 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:33.481396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:33.481410 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:33.481521 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.481527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.482251 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.482254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.483034 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:33.483045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.483714 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.483718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:33.483809 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.483845 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:57:33.483911 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:57:33.483987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":36,"thread_start_us":86,"threads_started":1}
I20260620 08:57:33.484838 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631c4da0ce6c45e2806f440b2911e1dc" candidate_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b016d74c13c44108be60bc9829b29a09" is_pre_election: true
I20260620 08:57:33.484861 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631c4da0ce6c45e2806f440b2911e1dc" candidate_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dcf0338b164a9aa916defcd8317765" is_pre_election: true
I20260620 08:57:33.485110 15941 raft_consensus.cc:2454] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cdefe194f6c4a1ba4ad6296af8ea0e9 in term 0.
I20260620 08:57:33.485105 16209 raft_consensus.cc:2454] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cdefe194f6c4a1ba4ad6296af8ea0e9 in term 0.
I20260620 08:57:33.485560 16011 leader_election.cc:302] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765, 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09; no voters:
I20260620 08:57:33.485818 16280 raft_consensus.cc:2781] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:57:33.485893 16280 raft_consensus.cc:482] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:57:33.485922 16280 raft_consensus.cc:3037] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:33.487063 16280 raft_consensus.cc:504] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.487424 16280 leader_election.cc:288] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [CANDIDATE]: Term 1 election: Requested vote from peers 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839), b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225)
W20260620 08:57:33.487753 16122 tablet.cc:2369] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:33.487766 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631c4da0ce6c45e2806f440b2911e1dc" candidate_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dcf0338b164a9aa916defcd8317765"
I20260620 08:57:33.487869 15941 raft_consensus.cc:3037] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:33.488889 15941 raft_consensus.cc:2454] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cdefe194f6c4a1ba4ad6296af8ea0e9 in term 1.
I20260620 08:57:33.489122 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631c4da0ce6c45e2806f440b2911e1dc" candidate_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b016d74c13c44108be60bc9829b29a09"
I20260620 08:57:33.489157 16011 leader_election.cc:302] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765, 4cdefe194f6c4a1ba4ad6296af8ea0e9; no voters:
I20260620 08:57:33.489239 16209 raft_consensus.cc:3037] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:33.489327 16280 raft_consensus.cc:2781] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:33.489548 16280 raft_consensus.cc:686] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4cdefe194f6c4a1ba4ad6296af8ea0e9, State: Running, Role: LEADER
I20260620 08:57:33.489995 16280 consensus_queue.cc:237] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:33.490238 16209 raft_consensus.cc:2454] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cdefe194f6c4a1ba4ad6296af8ea0e9 in term 1.
I20260620 08:57:33.492959 15795 catalog_manager.cc:5495] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } health_report { overall_health: UNKNOWN } } }
W20260620 08:57:33.520272 15988 tablet.cc:2369] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:33.551868 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.548652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.0.98.2:34817)
0620 08:57:33.548770 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:33.548774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:33.548787 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:57:33.548862 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:33.548865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:33.548919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:33.549007 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:33.549243 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.549752 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.550548 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:33.550800 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:33.550888 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:33.551096 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:33.551519 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:57:33.551636 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:33.551691 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 08:57:33.551743 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:57:33.557271 15941 raft_consensus.cc:1261] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Refusing update from remote peer 4cdefe194f6c4a1ba4ad6296af8ea0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:33.557271 16209 raft_consensus.cc:1261] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Refusing update from remote peer 4cdefe194f6c4a1ba4ad6296af8ea0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:33.557857 16282 consensus_queue.cc:1035] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:33.558025 16280 consensus_queue.cc:1035] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:33.573552 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:33.573712 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:33.575891 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Sending DeleteTablet for 3 replicas of tablet 631c4da0ce6c45e2806f440b2911e1dc
I20260620 08:57:33.576542 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35188
I20260620 08:57:33.576722 16189 tablet_service.cc:1514] Processing DeleteTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49788
I20260620 08:57:33.576894 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43246
I20260620 08:57:33.577143 16295 tablet_replica.cc:330] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: stopping tablet replica
I20260620 08:57:33.577200 16296 tablet_replica.cc:330] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: stopping tablet replica
I20260620 08:57:33.577421 16295 raft_consensus.cc:2229] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:57:33.577420 16296 raft_consensus.cc:2229] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:33.577507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15858
I20260620 08:57:33.578361 16295 raft_consensus.cc:2258] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:33.579276 16296 raft_consensus.cc:2258] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:33.580058 16295 ts_tablet_manager.cc:1905] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:33.580631 16296 ts_tablet_manager.cc:1905] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:33.582762 16295 ts_tablet_manager.cc:1918] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:33.582937 16295 log.cc:1196] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/631c4da0ce6c45e2806f440b2911e1dc
I20260620 08:57:33.583142 16296 ts_tablet_manager.cc:1918] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:33.583288 16295 ts_tablet_manager.cc:1939] T 631c4da0ce6c45e2806f440b2911e1dc P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting consensus metadata
I20260620 08:57:33.583298 16296 log.cc:1196] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/631c4da0ce6c45e2806f440b2911e1dc
I20260620 08:57:33.583577 16296 ts_tablet_manager.cc:1939] T 631c4da0ce6c45e2806f440b2911e1dc P b016d74c13c44108be60bc9829b29a09: Deleting consensus metadata
I20260620 08:57:33.584470 15782 catalog_manager.cc:4841] TS 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817): tablet 631c4da0ce6c45e2806f440b2911e1dc (table test-table [id=d4e7c0aed1d74067803d09365f651358]) successfully deleted
W20260620 08:57:33.584558 15783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41839 (error 108)
I20260620 08:57:33.585510 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:33.585232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41839 (local address 127.0.0.1:43262)
0620 08:57:33.585349 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:33.585406 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:57:33.585544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15992
W20260620 08:57:33.586694 15783 catalog_manager.cc:4584] TS 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 631c4da0ce6c45e2806f440b2911e1dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41839: connect: Connection refused (error 111)
I20260620 08:57:33.587587 15780 catalog_manager.cc:4841] TS b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): tablet 631c4da0ce6c45e2806f440b2911e1dc (table test-table [id=d4e7c0aed1d74067803d09365f651358]) successfully deleted
I20260620 08:57:33.590852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 16126
I20260620 08:57:33.595275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 15764
I20260620 08:57:53.600875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--webserver_interface=127.0.98.62
--webserver_port=40811
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:57:53.712423 16299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:53.712698 16299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:53.712738 16299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:53.716785 16299 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:57:53.716848 16299 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:57:53.716863 16299 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:57:53.716876 16299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:57:53.716892 16299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:53.716909 16299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:53.716925 16299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:57:53.716939 16299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:57:53.722307 16299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:53.723356 16299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:53.724395 16299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:53.737970 16308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:53.738060 16305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:53.738060 16306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:53.738392 16299 server_base.cc:1034] running on GCE node
I20260620 08:57:53.739018 16299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:53.740244 16299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:53.741456 16299 hybrid_clock.cc:648] HybridClock initialized: now 1781945873741430 us; error 44 us; skew 500 ppm
Jun 20 08:57:53 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:53.747835 16299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:57:53.748839 16299 webserver.cc:460] Webserver started at http://127.0.98.62:40811/ using document root <none> and password file <none>
I20260620 08:57:53.749396 16299 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:53.749487 16299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:53.753551 16299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:57:53.756019 16315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:53.757424 16299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 08:57:53.757591 16299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "87d5c93b3ddf4572bb505bdadb7d73cb"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "5985c8ee0954319afd21f03ed0f6b186"
server_key_iv: "4aa7b699e5c34144c12752f862f9b1fb"
server_key_version: "encryptionkey@0"
I20260620 08:57:53.758180 16299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:53.775156 16299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:53.778239 16299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:53.778417 16299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:53.786015 16299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45763
I20260620 08:57:53.786353 16367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45763 every 8 connection(s)
I20260620 08:57:53.787103 16299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:57:53.789028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16299
I20260620 08:57:53.789625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41839
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45763
--webserver_port=35421
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:53.793386 16368 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:57:53.795507 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap starting.
I20260620 08:57:53.816064 16368 log.cc:824] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:53.826776 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:53.827250 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap complete.
I20260620 08:57:53.832041 16368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:53.832592 16368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d5c93b3ddf4572bb505bdadb7d73cb, State: Initialized, Role: FOLLOWER
I20260620 08:57:53.833206 16368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:53.833365 16368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:57:53.833446 16368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:57:53.833549 16368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:57:53.835934 16368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:53.836315 16368 leader_election.cc:302] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d5c93b3ddf4572bb505bdadb7d73cb; no voters:
I20260620 08:57:53.836704 16368 leader_election.cc:288] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:57:53.836925 16372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:57:53.837368 16372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 LEADER]: Becoming Leader. State: Replica: 87d5c93b3ddf4572bb505bdadb7d73cb, State: Running, Role: LEADER
I20260620 08:57:53.837849 16372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:57:53.838124 16368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:57:53.839572 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d5c93b3ddf4572bb505bdadb7d73cb. Latest consensus state: current_term: 2 leader_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260620 08:57:53.839751 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:53.840087 16373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260620 08:57:53.840168 16373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: This master's current role is: LEADER
I20260620 08:57:53.844095 16388 catalog_manager.cc:1259] Loaded cluster ID: b4b0ba4410a649f3ba2bb2e916d6dfa1
I20260620 08:57:53.844182 16388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: loading cluster ID for follower catalog manager: success
I20260620 08:57:53.846282 16388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: acquiring CA information for follower catalog manager: success
I20260620 08:57:53.847716 16388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:57:53.847848 16383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:57:53.850257 16383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:57:53.850998 16383 catalog_manager.cc:1259] Loaded cluster ID: b4b0ba4410a649f3ba2bb2e916d6dfa1
I20260620 08:57:53.851052 16383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:57:53.851754 16383 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:57:53.852169 16383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Loaded TSK: 0
I20260620 08:57:53.867306 16383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Generated new TSK 1
I20260620 08:57:53.867705 16383 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:57:53.916076 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:53.916325 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:53.916386 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:53.920428 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:53.920492 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:53.920575 16370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:57:53.927590 16370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35421
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:53.928720 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:53.929768 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:53.939659 16396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:53.939683 16395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:53.939759 16398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:53.939958 16370 server_base.cc:1034] running on GCE node
I20260620 08:57:53.940434 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:53.941156 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:53.942350 16370 hybrid_clock.cc:648] HybridClock initialized: now 1781945873942331 us; error 31 us; skew 500 ppm
Jun 20 08:57:53 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:53.946063 16370 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:57:53.947075 16370 webserver.cc:460] Webserver started at http://127.0.98.1:35421/ using document root <none> and password file <none>
I20260620 08:57:53.947599 16370 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:53.947710 16370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:53.951341 16370 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:57:53.953456 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:53.954717 16370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 08:57:53.954838 16370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "27dcf0338b164a9aa916defcd8317765"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "1ba84319372bcd2d7ddb7065212c31c8"
server_key_iv: "0ed9563cd683e51634de604f4521fba5"
server_key_version: "encryptionkey@0"
I20260620 08:57:53.955354 16370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:53.978866 16370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:53.982049 16370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:53.982331 16370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:53.983024 16370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:53.984226 16412 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:57:53.989607 16370 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:57:53.989681 16370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260620 08:57:53.989753 16370 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:57:53.991470 16370 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:57:53.991514 16370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 08:57:53.991888 16412 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Bootstrap starting.
I20260620 08:57:54.001351 16370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41839
I20260620 08:57:54.001631 16519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41839 every 8 connection(s)
I20260620 08:57:54.002351 16370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:57:54.007699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16370
I20260620 08:57:54.008242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34817
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45763
--webserver_port=38031
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:57:54.012681 16412 log.cc:824] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:57:54 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:54.021262 16412 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:57:54.021665 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.004309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:36589)
0620 08:57:54.005268 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.005310 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.006096 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.007129 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.007136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.007568 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:54.008103 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.008118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.009963 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.009968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.011841 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.011857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.012897 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.012900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.013053 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.013617 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:54.013641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:54.015531 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:54.018038 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.018045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.018059 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.018402 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.018816 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.018819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.018821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.018999 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.019804 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:54.019811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:54.020136 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.020347 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.020719 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":122,"threads_started":1}
I20260620 08:57:54.021690 16412 tablet_bootstrap.cc:492] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Bootstrap complete.
I20260620 08:57:54.021688 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.004280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36589 (local address 127.0.98.62:45763)
0620 08:57:54.005009 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.005031 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.005427 (+ 396us) server_negotiation.cc:407] Connection header received
0620 08:57:54.006475 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.006485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.006760 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.007056 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:54.008274 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.009581 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.012009 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.012552 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.012702 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.015781 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:54.015788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:54.015804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:54.015839 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:54.017897 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.018544 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.018550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.018555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.018708 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.019113 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.019116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.019118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.019680 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:54.019816 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.020503 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.020799 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":132,"threads_started":1}
I20260620 08:57:54.022821 16412 ts_tablet_manager.cc:1397] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.012s
I20260620 08:57:54.024531 16520 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260620 08:57:54.024742 16520 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:54.025255 16520 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260620 08:57:54.025676 16412 raft_consensus.cc:348] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:54.026160 16412 raft_consensus.cc:729] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dcf0338b164a9aa916defcd8317765, State: Initialized, Role: FOLLOWER
I20260620 08:57:54.026597 16412 consensus_queue.cc:260] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } } peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } }
I20260620 08:57:54.027664 16412 ts_tablet_manager.cc:1428] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:57:54.027913 16332 ts_manager.cc:194] Registered new tserver with Master: 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839)
I20260620 08:57:54.031430 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36589
I20260620 08:57:54.034009 16520 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
Jun 20 08:57:54 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:57:54.041242 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.030633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41839 (local address 127.0.0.1:50472)
0620 08:57:54.031419 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.031462 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.031715 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.032378 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.032383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.032832 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:54.032932 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.032939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.033900 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.033904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.034647 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.034655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.035328 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.035343 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.035448 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.036067 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:54.036079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:54.037548 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:54.039644 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.039653 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.039658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.039927 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.040221 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.040223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.040229 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.040304 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.040774 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:54.040779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:54.040872 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.040974 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.041115 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":110,"threads_started":1}
I20260620 08:57:54.041594 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.030700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.98.1:41839)
0620 08:57:54.030979 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.030993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.031636 (+ 643us) server_negotiation.cc:407] Connection header received
0620 08:57:54.031871 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.031876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.032090 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.032214 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:54.033052 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.033793 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.034767 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.035237 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.035340 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.037686 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:54.037695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:54.037707 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:54.037732 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:54.039505 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.040037 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.040042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.040044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.040134 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.040393 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.040395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.040397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.040676 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:54.040791 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.041080 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.041237 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260620 08:57:54.042481 16454 tablet_service.cc:1514] Processing DeleteTablet for tablet 631c4da0ce6c45e2806f440b2911e1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50472
I20260620 08:57:54.043306 16529 tablet_replica.cc:330] stopping tablet replica
I20260620 08:57:54.043645 16529 raft_consensus.cc:2229] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:54.043903 16529 raft_consensus.cc:2258] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:54.044967 16529 ts_tablet_manager.cc:1905] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:54.049387 16529 ts_tablet_manager.cc:1918] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:54.049480 16529 log.cc:1196] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal/wals/631c4da0ce6c45e2806f440b2911e1dc
I20260620 08:57:54.049729 16529 ts_tablet_manager.cc:1939] T 631c4da0ce6c45e2806f440b2911e1dc P 27dcf0338b164a9aa916defcd8317765: Deleting consensus metadata
I20260620 08:57:54.051846 16319 catalog_manager.cc:4841] TS 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839): tablet 631c4da0ce6c45e2806f440b2911e1dc (table test-table [id=d4e7c0aed1d74067803d09365f651358]) successfully deleted
W20260620 08:57:54.125897 16524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:54.126143 16524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:54.126190 16524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:54.130113 16524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:54.130168 16524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:54.130223 16524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:57:54.135293 16524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38031
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:54.136397 16524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:54.137297 16524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:54.146333 16537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:54.146384 16535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:54.146382 16534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:54.146667 16524 server_base.cc:1034] running on GCE node
I20260620 08:57:54.147086 16524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:54.147727 16524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:54.148900 16524 hybrid_clock.cc:648] HybridClock initialized: now 1781945874148871 us; error 42 us; skew 500 ppm
Jun 20 08:57:54 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:54.151440 16524 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:57:54.152292 16524 webserver.cc:460] Webserver started at http://127.0.98.2:38031/ using document root <none> and password file <none>
I20260620 08:57:54.152803 16524 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:54.152880 16524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:54.156630 16524 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:54.158749 16544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:54.160051 16524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:57:54.160171 16524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "aeb1a1bb184a302b0b6a166fb4a12145"
server_key_iv: "f6e37aefc57e7d3d85b838698393ab97"
server_key_version: "encryptionkey@0"
I20260620 08:57:54.160710 16524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:54.187134 16524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:54.190395 16524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:54.190584 16524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:54.191181 16524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:54.192037 16524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:54.192085 16524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:54.192139 16524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:54.192168 16524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:54.200871 16524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34817
I20260620 08:57:54.200892 16657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34817 every 8 connection(s)
I20260620 08:57:54.201798 16524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:57:54.207249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16524
I20260620 08:57:54.207677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33225
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45763
--webserver_port=46699
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:57:54 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:54.215185 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.203571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44829 (local address 127.0.98.62:45763)
0620 08:57:54.203741 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.203745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.204705 (+ 960us) server_negotiation.cc:407] Connection header received
0620 08:57:54.205675 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.205678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.205732 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.205805 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:54.206919 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.207482 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.208670 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.208898 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.208983 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.211543 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:54.211546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:54.211548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:54.211572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:54.213009 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.213596 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.213598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.213599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.213671 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.214051 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.214055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.214057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.214237 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:54.214331 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.214938 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.215062 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 08:57:54.216286 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.203910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:44829)
0620 08:57:54.204555 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.204593 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.205457 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.205978 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.205985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.206332 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:54.206780 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.206792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.207597 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.207600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.208549 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.208558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.209030 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.209034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.209193 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.209726 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:54.209742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:54.211349 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:54.213141 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.213147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.213162 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.213491 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.213752 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.213755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.213757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.213930 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.214348 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:54.214355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:54.214676 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.214890 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.215239 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":102,"threads_started":1}
I20260620 08:57:54.217507 16658 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260620 08:57:54.217729 16658 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:54.218176 16658 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260620 08:57:54.219118 16332 ts_manager.cc:194] Registered new tserver with Master: 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
I20260620 08:57:54.219705 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44829
W20260620 08:57:54.317866 16661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:57:54.318126 16661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:57:54.318182 16661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:57:54.322134 16661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:57:54.322190 16661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:57:54.322266 16661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:57:54.327446 16661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46699
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:57:54.328429 16661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:57:54.329387 16661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:57:54.338881 16666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:54.338960 16667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:57:54.338886 16669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:57:54.339003 16661 server_base.cc:1034] running on GCE node
I20260620 08:57:54.339799 16661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:57:54.340480 16661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:57:54.341653 16661 hybrid_clock.cc:648] HybridClock initialized: now 1781945874341633 us; error 30 us; skew 500 ppm
Jun 20 08:57:54 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:57:54.344527 16661 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:57:54.345360 16661 webserver.cc:460] Webserver started at http://127.0.98.3:46699/ using document root <none> and password file <none>
I20260620 08:57:54.345877 16661 fs_manager.cc:359] Metadata directory not provided
I20260620 08:57:54.345953 16661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:57:54.349723 16661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 08:57:54.351797 16676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:54.353073 16661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:57:54.353185 16661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "b016d74c13c44108be60bc9829b29a09"
format_stamp: "Formatted at 2026-06-20 08:57:33 on dist-test-slave-fb1v"
server_key: "3d1f9e022d5cb06829850867f0745e73"
server_key_iv: "61c610aa7faa9d2c06c273e12c9f2974"
server_key_version: "encryptionkey@0"
I20260620 08:57:54.353691 16661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:57:54.371918 16661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:57:54.374688 16661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:57:54.374855 16661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:57:54.375416 16661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:57:54.376240 16661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:57:54.376286 16661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:54.376339 16661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:57:54.376365 16661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:57:54.384578 16661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33225
I20260620 08:57:54.384649 16789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33225 every 8 connection(s)
I20260620 08:57:54.385512 16661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:57:54.394066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16661
Jun 20 08:57:54 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:57:54.398667 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.387223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51283 (local address 127.0.98.62:45763)
0620 08:57:54.387363 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.387368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.388338 (+ 970us) server_negotiation.cc:407] Connection header received
0620 08:57:54.389151 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.389154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.389198 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.389259 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:54.390446 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.391203 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.392390 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.392724 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.392825 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.395410 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:54.395413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:54.395415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:54.395436 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:54.396760 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.397225 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.397229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.397231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.397303 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.397609 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.397612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.397613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.397780 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:54.397864 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.398381 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.398505 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:57:54.399444 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.387557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:51283)
0620 08:57:54.388195 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.388233 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.388971 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.389404 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.389410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.389758 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:57:54.390199 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.390212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.391345 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.391348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.392270 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.392280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.392879 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.392882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.393032 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.393637 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:57:54.393654 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:57:54.395243 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:57:54.396870 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.396876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.396890 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.397128 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.397387 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:57:54.397390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:57:54.397392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:57:54.397522 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:57:54.397883 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:57:54.397889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:57:54.398119 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.398317 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.398590 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":109,"threads_started":1}
I20260620 08:57:54.400614 16790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260620 08:57:54.400828 16790 heartbeater.cc:464] Registering TS with master...
I20260620 08:57:54.401283 16790 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260620 08:57:54.402257 16332 ts_manager.cc:194] Registered new tserver with Master: b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225)
I20260620 08:57:54.402453 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.394718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.98.62:45763)
0620 08:57:54.394982 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.394985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.395235 (+ 250us) server_negotiation.cc:407] Connection header received
0620 08:57:54.395350 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.395353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.395396 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.395470 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:54.396217 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.396729 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.397552 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.397773 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.397857 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.399193 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:54.399196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:54.399198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:54.399217 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:54.400941 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.401303 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.401306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.401308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.401391 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.401635 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.401639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.401640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.401851 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:54.402024 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.402208 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.402319 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260620 08:57:54.402908 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51283
I20260620 08:57:54.404075 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:57:54.408049 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.404830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.98.62:45763)
0620 08:57:54.404946 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.404950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.404962 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:57:54.405022 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.405025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.405117 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.405198 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:54.405687 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.406190 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.407010 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.407230 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.407314 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.407576 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:54.407722 (+ 146us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 08:57:54.407932 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44396: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 08:57:54.408108 16793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44396: token expired
W20260620 08:57:54.408473 16261 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45763)
I20260620 08:57:54.415118 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.408781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.0.98.62:45763)
0620 08:57:54.408900 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.408904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.408916 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:57:54.409013 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.409016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.409061 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.409124 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:57:54.409689 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.410156 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.410913 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.411132 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.411211 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.412393 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:57:54.412397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:57:54.412399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:57:54.412423 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:57:54.413689 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.414061 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.414064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.414066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.414141 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:57:54.414377 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:57:54.414380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:57:54.414382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:57:54.414541 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:57:54.414651 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.414870 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.414971 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:57:54.417393 16332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:57:54.420573 16332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:57:54.428194 16454 tablet_service.cc:1467] Processing CreateTablet for tablet bce8039d35a54df9bee325a2e02e623e (DEFAULT_TABLE table=test-table [id=4c2870691bd74a0ebf2ab28571438cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:54.428828 16454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce8039d35a54df9bee325a2e02e623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:54.431445 16412 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Bootstrap starting.
I20260620 08:57:54.431810 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.426788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33225 (local address 127.0.0.1:36614)
0620 08:57:54.427173 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.427188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.427266 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.427984 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.427987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.428009 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:54.428123 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.428130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.429455 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.429459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.430423 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.430433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.431355 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.431359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.431464 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.431510 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.431585 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.431685 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":3}
I20260620 08:57:54.432169 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.426966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.0.98.3:33225)
0620 08:57:54.427459 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.427470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.427497 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:57:54.427578 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.427583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.427820 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.427986 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:54.428215 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.429285 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.430567 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.431214 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.431322 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.431464 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.431713 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.431816 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":147,"threads_started":1}
I20260620 08:57:54.432322 16412 tablet_bootstrap.cc:654] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:54.433552 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.427069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34817 (local address 127.0.0.1:37102)
0620 08:57:54.427301 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.427312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.427378 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.428327 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.428329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.428344 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:54.428414 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.428420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.429718 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.429721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.430838 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.430848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.432840 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.432843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.432933 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.432954 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.433039 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.433153 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":27,"thread_start_us":90,"threads_started":1}
I20260620 08:57:54.433873 16412 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: No bootstrap required, opened a new log
I20260620 08:57:54.433964 16412 ts_tablet_manager.cc:1397] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:57:54.434015 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.427387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.0.98.2:34817)
0620 08:57:54.427709 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.427725 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.427746 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:57:54.427812 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.427815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.427982 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.428143 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:54.428531 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.429603 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.430966 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.432556 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.432655 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.432780 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.433794 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.433882 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260620 08:57:54.434392 16412 raft_consensus.cc:348] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.434513 16412 raft_consensus.cc:374] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:54.434551 16412 raft_consensus.cc:729] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dcf0338b164a9aa916defcd8317765, State: Initialized, Role: FOLLOWER
I20260620 08:57:54.434679 16412 consensus_queue.cc:260] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.435405 16412 ts_tablet_manager.cc:1428] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:57:54.435429 16724 tablet_service.cc:1467] Processing CreateTablet for tablet bce8039d35a54df9bee325a2e02e623e (DEFAULT_TABLE table=test-table [id=4c2870691bd74a0ebf2ab28571438cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:54.436388 16724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce8039d35a54df9bee325a2e02e623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:54.437430 16592 tablet_service.cc:1467] Processing CreateTablet for tablet bce8039d35a54df9bee325a2e02e623e (DEFAULT_TABLE table=test-table [id=4c2870691bd74a0ebf2ab28571438cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:57:54.438375 16592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce8039d35a54df9bee325a2e02e623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:57:54.443185 16803 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Bootstrap starting.
I20260620 08:57:54.444757 16804 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Bootstrap starting.
I20260620 08:57:54.445580 16803 tablet_bootstrap.cc:654] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:54.446259 16803 log.cc:824] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:54.446763 16804 tablet_bootstrap.cc:654] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Neither blocks nor log segments found. Creating new log.
I20260620 08:57:54.447384 16804 log.cc:824] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:57:54.447981 16803 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: No bootstrap required, opened a new log
I20260620 08:57:54.448166 16803 ts_tablet_manager.cc:1397] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:57:54.448995 16804 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: No bootstrap required, opened a new log
I20260620 08:57:54.449177 16804 ts_tablet_manager.cc:1397] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 08:57:54.451035 16803 raft_consensus.cc:348] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.451249 16803 raft_consensus.cc:374] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:54.451282 16803 raft_consensus.cc:729] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b016d74c13c44108be60bc9829b29a09, State: Initialized, Role: FOLLOWER
I20260620 08:57:54.451730 16803 consensus_queue.cc:260] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.451876 16804 raft_consensus.cc:348] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.452086 16804 raft_consensus.cc:374] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:57:54.452135 16804 raft_consensus.cc:729] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdefe194f6c4a1ba4ad6296af8ea0e9, State: Initialized, Role: FOLLOWER
I20260620 08:57:54.452320 16790 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260620 08:57:54.452510 16804 consensus_queue.cc:260] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.452683 16803 ts_tablet_manager.cc:1428] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:57:54.453078 16658 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260620 08:57:54.453528 16804 ts_tablet_manager.cc:1428] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260620 08:57:54.494626 16526 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:57:54.494840 16526 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.495975 16526 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
I20260620 08:57:54.501896 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.496740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54625 (local address 127.0.98.3:33225)
0620 08:57:54.496910 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.496915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.496935 (+ 20us) server_negotiation.cc:407] Connection header received
0620 08:57:54.497561 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.497565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.497639 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.497715 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:54.498057 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.498818 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.500582 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.501217 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.501332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.501401 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.501696 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.501767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:57:54.501896 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.497025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41053 (local address 127.0.98.2:34817)
0620 08:57:54.497225 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.497229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.497755 (+ 526us) server_negotiation.cc:407] Connection header received
0620 08:57:54.497871 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.497874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.497917 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.497977 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:57:54.498219 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.498917 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.500582 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.501217 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.501332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.501401 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.501696 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.501768 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 08:57:54.501931 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.496515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33225 (local address 127.0.98.1:54625)
0620 08:57:54.496801 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.496824 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.497180 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.497752 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.497755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.497816 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:54.497936 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.497941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.498950 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.498953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.500446 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.500458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.501360 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.501364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.501516 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.501605 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.501675 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.501772 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":177,"spinlock_wait_cycles":21248}
I20260620 08:57:54.501950 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.496523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34817 (local address 127.0.98.1:41053)
0620 08:57:54.497638 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.497651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:57:54.497769 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:57:54.498014 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:57:54.498017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:57:54.498034 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:57:54.498124 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.498128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.499037 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.499040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.500449 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:57:54.500458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.501362 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.501365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:57:54.501542 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.501596 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 08:57:54.501675 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:57:54.501772 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":856,"thread_start_us":88,"threads_started":1}
I20260620 08:57:54.502544 16612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce8039d35a54df9bee325a2e02e623e" candidate_uuid: "27dcf0338b164a9aa916defcd8317765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" is_pre_election: true
I20260620 08:57:54.502627 16744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce8039d35a54df9bee325a2e02e623e" candidate_uuid: "27dcf0338b164a9aa916defcd8317765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b016d74c13c44108be60bc9829b29a09" is_pre_election: true
I20260620 08:57:54.502861 16612 raft_consensus.cc:2454] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27dcf0338b164a9aa916defcd8317765 in term 0.
I20260620 08:57:54.502918 16744 raft_consensus.cc:2454] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27dcf0338b164a9aa916defcd8317765 in term 0.
I20260620 08:57:54.503360 16408 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765, 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09; no voters:
W20260620 08:57:54.503657 16521 tablet.cc:2369] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:57:54.503676 16526 raft_consensus.cc:2781] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:57:54.503787 16526 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:57:54.503836 16526 raft_consensus.cc:3037] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:54.504848 16526 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.505174 16526 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 1 election: Requested vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
I20260620 08:57:54.505481 16744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce8039d35a54df9bee325a2e02e623e" candidate_uuid: "27dcf0338b164a9aa916defcd8317765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b016d74c13c44108be60bc9829b29a09"
I20260620 08:57:54.505512 16612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce8039d35a54df9bee325a2e02e623e" candidate_uuid: "27dcf0338b164a9aa916defcd8317765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9"
I20260620 08:57:54.505607 16612 raft_consensus.cc:3037] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:54.505601 16744 raft_consensus.cc:3037] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:57:54.506789 16612 raft_consensus.cc:2454] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27dcf0338b164a9aa916defcd8317765 in term 1.
I20260620 08:57:54.506789 16744 raft_consensus.cc:2454] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27dcf0338b164a9aa916defcd8317765 in term 1.
I20260620 08:57:54.507058 16406 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765, b016d74c13c44108be60bc9829b29a09; no voters:
I20260620 08:57:54.507227 16526 raft_consensus.cc:2781] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:57:54.507643 16526 raft_consensus.cc:686] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 LEADER]: Becoming Leader. State: Replica: 27dcf0338b164a9aa916defcd8317765, State: Running, Role: LEADER
I20260620 08:57:54.507902 16526 consensus_queue.cc:237] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:57:54.510788 16331 catalog_manager.cc:5495] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27dcf0338b164a9aa916defcd8317765 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "27dcf0338b164a9aa916defcd8317765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } health_report { overall_health: HEALTHY } } }
I20260620 08:57:54.531482 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.528154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.0.98.1:41839)
0620 08:57:54.528290 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:57:54.528297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:57:54.528310 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:57:54.528385 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:57:54.528387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:57:54.528440 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:57:54.528535 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:57:54.528778 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.529303 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.530099 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:57:54.530316 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:57:54.530387 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:57:54.530661 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:57:54.531060 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:57:54.531141 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:57:54.531313 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 08:57:54.531375 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 08:57:54.535425 16612 raft_consensus.cc:1261] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Refusing update from remote peer 27dcf0338b164a9aa916defcd8317765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:54.535425 16744 raft_consensus.cc:1261] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Refusing update from remote peer 27dcf0338b164a9aa916defcd8317765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:57:54.536144 16526 consensus_queue.cc:1035] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:54.536327 16810 consensus_queue.cc:1035] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:57:54.544096 16815 mvcc.cc:205] Tried to move back new op lower bound from 7298850302090670080 to 7298850301986521088. Current Snapshot: MvccSnapshot[applied={T|T < 7298850302090670080}]
I20260620 08:57:54.546140 16814 mvcc.cc:205] Tried to move back new op lower bound from 7298850302090670080 to 7298850301986521088. Current Snapshot: MvccSnapshot[applied={T|T < 7298850302090670080}]
I20260620 08:57:54.550766 16330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:54.550928 16330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:57:54.553225 16330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Sending DeleteTablet for 3 replicas of tablet bce8039d35a54df9bee325a2e02e623e
I20260620 08:57:54.553850 16724 tablet_service.cc:1514] Processing DeleteTablet for tablet bce8039d35a54df9bee325a2e02e623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36614
I20260620 08:57:54.553889 16592 tablet_service.cc:1514] Processing DeleteTablet for tablet bce8039d35a54df9bee325a2e02e623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37102
I20260620 08:57:54.553938 16454 tablet_service.cc:1514] Processing DeleteTablet for tablet bce8039d35a54df9bee325a2e02e623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:57:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50472
I20260620 08:57:54.554292 16823 tablet_replica.cc:330] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: stopping tablet replica
I20260620 08:57:54.554388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 16370
I20260620 08:57:54.554414 16823 raft_consensus.cc:2229] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:57:54.554687 16823 raft_consensus.cc:2258] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:54.556629 16825 tablet_replica.cc:330] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: stopping tablet replica
I20260620 08:57:54.556907 16825 raft_consensus.cc:2229] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:54.557171 16825 raft_consensus.cc:2258] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:54.557832 16824 tablet_replica.cc:330] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: stopping tablet replica
I20260620 08:57:54.558105 16824 raft_consensus.cc:2229] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:57:54.558370 16824 raft_consensus.cc:2258] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:57:54.558509 16825 ts_tablet_manager.cc:1905] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:57:54.559561 16824 ts_tablet_manager.cc:1905] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:57:54.560950 16319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41839 (error 108)
I20260620 08:57:54.561134 16825 ts_tablet_manager.cc:1918] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:54.561211 16825 log.cc:1196] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/bce8039d35a54df9bee325a2e02e623e
I20260620 08:57:54.561314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 16524
I20260620 08:57:54.561518 16825 ts_tablet_manager.cc:1939] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting consensus metadata
I20260620 08:57:54.562659 16824 ts_tablet_manager.cc:1918] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:57:54.562731 16824 log.cc:1196] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/bce8039d35a54df9bee325a2e02e623e
I20260620 08:57:54.562994 16824 ts_tablet_manager.cc:1939] T bce8039d35a54df9bee325a2e02e623e P b016d74c13c44108be60bc9829b29a09: Deleting consensus metadata
I20260620 08:57:54.564038 16316 catalog_manager.cc:4841] TS b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): tablet bce8039d35a54df9bee325a2e02e623e (table test-table [id=4c2870691bd74a0ebf2ab28571438cd9]) successfully deleted
I20260620 08:57:54.566860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 16661
I20260620 08:57:54.567589 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.567022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41839 (local address 127.0.0.1:50490)
0620 08:57:54.567433 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.567507 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260620 08:57:54.567816 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:57:54.567091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34817 (local address 127.0.0.1:37112)
0620 08:57:54.567712 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 08:57:54.567761 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553}
W20260620 08:57:54.567911 16319 catalog_manager.cc:4584] TS 27dcf0338b164a9aa916defcd8317765 (127.0.98.1:41839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bce8039d35a54df9bee325a2e02e623e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41839: connect: Connection refused (error 111)
I20260620 08:57:54.573244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 16299
I20260620 08:58:14.578867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--webserver_interface=127.0.98.62
--webserver_port=40811
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:14.707690 16827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:14.708060 16827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:14.708117 16827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:14.712406 16827 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:58:14.712477 16827 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:58:14.712492 16827 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:58:14.712505 16827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:14.712525 16827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:14.712543 16827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:14.712558 16827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:14.712574 16827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:14.718569 16827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:14.720016 16827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:14.721278 16827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:14.731189 16833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:14.731191 16832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:14.731350 16827 server_base.cc:1034] running on GCE node
W20260620 08:58:14.731191 16835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:14.732370 16827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:14.733878 16827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:14.735073 16827 hybrid_clock.cc:648] HybridClock initialized: now 1781945894735056 us; error 32 us; skew 500 ppm
Jun 20 08:58:14 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:14.741353 16827 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:58:14.742573 16827 webserver.cc:460] Webserver started at http://127.0.98.62:40811/ using document root <none> and password file <none>
I20260620 08:58:14.743247 16827 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:14.743320 16827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:14.748131 16827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:14.750635 16842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:14.752138 16827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:14.752269 16827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "87d5c93b3ddf4572bb505bdadb7d73cb"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "5985c8ee0954319afd21f03ed0f6b186"
server_key_iv: "4aa7b699e5c34144c12752f862f9b1fb"
server_key_version: "encryptionkey@0"
I20260620 08:58:14.753119 16827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:14.772708 16827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:14.776887 16827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:14.777125 16827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:14.786062 16827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45763
I20260620 08:58:14.786350 16894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45763 every 8 connection(s)
I20260620 08:58:14.787514 16827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:14.793236 16895 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:58:14.795469 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap starting.
I20260620 08:58:14.797477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16827
I20260620 08:58:14.798026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41839
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45763
--webserver_port=35421
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:58:14.816954 16895 log.cc:824] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:14.829726 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:58:14.830194 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Bootstrap complete.
I20260620 08:58:14.834903 16895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:58:14.835374 16895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d5c93b3ddf4572bb505bdadb7d73cb, State: Initialized, Role: FOLLOWER
I20260620 08:58:14.835987 16895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:58:14.836134 16895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:14.836210 16895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:14.836311 16895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 2 FOLLOWER]: Advancing to term 3
I20260620 08:58:14.838914 16895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:58:14.839380 16895 leader_election.cc:302] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d5c93b3ddf4572bb505bdadb7d73cb; no voters:
I20260620 08:58:14.839830 16895 leader_election.cc:288] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 08:58:14.839979 16899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 3 FOLLOWER]: Leader election won for term 3
I20260620 08:58:14.840215 16899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [term 3 LEADER]: Becoming Leader. State: Replica: 87d5c93b3ddf4572bb505bdadb7d73cb, State: Running, Role: LEADER
I20260620 08:58:14.840588 16899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260620 08:58:14.841434 16895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:14.842341 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260620 08:58:14.842373 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d5c93b3ddf4572bb505bdadb7d73cb. Latest consensus state: current_term: 3 leader_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d5c93b3ddf4572bb505bdadb7d73cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260620 08:58:14.842494 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:14.842494 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:14.842875 16907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:14.849637 16907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:14.850440 16907 catalog_manager.cc:1259] Loaded cluster ID: b4b0ba4410a649f3ba2bb2e916d6dfa1
I20260620 08:58:14.850513 16907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:14.853271 16907 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:14.854251 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Loaded TSK: 0
I20260620 08:58:14.854349 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87d5c93b3ddf4572bb505bdadb7d73cb: Loaded TSK: 1
F20260620 08:58:14.855268 16907 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f71879e19ad google::LogMessage::Fail() at ??:0
@ 0x7f71879e488a google::LogMessage::SendToLog() at ??:0
@ 0x7f71879e14de google::LogMessage::Flush() at ??:0
@ 0x7f71879e2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f718a6a518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f718a69df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f71923467bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f719232099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7192321178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f7192362df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f719238182b std::function<>::operator()() at ??:0
@ 0x7f7192320b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7192321f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f719231e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7192362640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7189c333d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7189c33c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7189c353f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7189c24512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f718b2686db start_thread at ??:0
@ 0x7f71864df61f clone at ??:0
W20260620 08:58:14.947867 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:14.948199 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:14.948248 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:14.953114 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:14.953184 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:14.953255 16897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:14.960078 16897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35421
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:14.961529 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:14.962815 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:14.979349 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:14.979351 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:14.979349 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:14.980767 16897 server_base.cc:1034] running on GCE node
I20260620 08:58:14.981530 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:14.982438 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:14.983875 16897 hybrid_clock.cc:648] HybridClock initialized: now 1781945894983852 us; error 50 us; skew 500 ppm
Jun 20 08:58:14 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:14.987596 16897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:58:14.988876 16897 webserver.cc:460] Webserver started at http://127.0.98.1:35421/ using document root <none> and password file <none>
I20260620 08:58:14.989615 16897 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:14.989701 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:14.995434 16897 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 08:58:14.998713 16952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:15.000617 16897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:58:15.000785 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "27dcf0338b164a9aa916defcd8317765"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "1ba84319372bcd2d7ddb7065212c31c8"
server_key_iv: "0ed9563cd683e51634de604f4521fba5"
server_key_version: "encryptionkey@0"
I20260620 08:58:15.001580 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:15.027819 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:15.032189 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:15.032430 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:15.033284 16897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:15.034896 16959 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:58:15.042441 16897 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:58:15.042570 16897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260620 08:58:15.042660 16897 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:58:15.044973 16897 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:58:15.045037 16897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 08:58:15.045439 16959 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Bootstrap starting.
I20260620 08:58:15.057116 16897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41839
I20260620 08:58:15.057441 17066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41839 every 8 connection(s)
I20260620 08:58:15.058524 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:15.068295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 16897
I20260620 08:58:15.068876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34817
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45763
--webserver_port=38031
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:58:15.071552 16959 log.cc:824] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:15.083858 16959 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:58:15.084451 16959 tablet_bootstrap.cc:492] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Bootstrap complete.
I20260620 08:58:15.085996 16959 ts_tablet_manager.cc:1397] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Time spent bootstrapping tablet: real 0.041s user 0.030s sys 0.008s
I20260620 08:58:15.090521 16959 raft_consensus.cc:348] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:15.091249 16959 raft_consensus.cc:729] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dcf0338b164a9aa916defcd8317765, State: Initialized, Role: FOLLOWER
I20260620 08:58:15.091976 16959 consensus_queue.cc:260] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:15.093319 16959 ts_tablet_manager.cc:1428] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
W20260620 08:58:15.138522 17069 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:58:15.061005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:37195)
0620 08:58:15.061596 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.061659 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:15.062642 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:15.137253 (+ 74611us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
W20260620 08:58:15.139191 17067 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45763 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45763: Client connection negotiation failed: client connection to 127.0.98.62:45763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:58:15.140152 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.139813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:44553)
0620 08:58:15.139963 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.140067 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 08:58:15.140820 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.140596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:33295)
0620 08:58:15.140705 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.140758 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 08:58:15.141083 17067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:58:15.205004 17070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:15.205394 17070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:15.205463 17070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:15.210008 17070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:15.210103 17070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:15.210201 17070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:15.216367 17070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38031
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:15.217895 17070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:15.219223 17070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:15.231025 17078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:15.231015 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:15.231104 17070 server_base.cc:1034] running on GCE node
W20260620 08:58:15.231015 17080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:15.231835 17070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:15.232717 17070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:15.233894 17070 hybrid_clock.cc:648] HybridClock initialized: now 1781945895233862 us; error 45 us; skew 500 ppm
Jun 20 08:58:15 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:15.237151 17070 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:58:15.238318 17070 webserver.cc:460] Webserver started at http://127.0.98.2:38031/ using document root <none> and password file <none>
I20260620 08:58:15.239017 17070 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:15.239096 17070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:15.243893 17070 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:15.246491 17087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:15.248132 17070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:15.248273 17070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9"
format_stamp: "Formatted at 2026-06-20 08:57:32 on dist-test-slave-fb1v"
server_key: "aeb1a1bb184a302b0b6a166fb4a12145"
server_key_iv: "f6e37aefc57e7d3d85b838698393ab97"
server_key_version: "encryptionkey@0"
I20260620 08:58:15.249044 17070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:15.264583 17070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:15.268515 17070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:15.268731 17070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:15.269486 17070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:15.270780 17094 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 08:58:15.275534 17094 ts_tablet_manager.cc:1852] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 08:58:15.276055 17094 ts_tablet_manager.cc:1905] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:15.281219 17094 ts_tablet_manager.cc:1918] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:15.281423 17094 ts_tablet_manager.cc:1939] T bce8039d35a54df9bee325a2e02e623e P 4cdefe194f6c4a1ba4ad6296af8ea0e9: Deleting consensus metadata
I20260620 08:58:15.282058 17070 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 08:58:15.282140 17070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.000s
I20260620 08:58:15.282233 17070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:15.282258 17070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:15.293334 17070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34817
I20260620 08:58:15.293395 17201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34817 every 8 connection(s)
I20260620 08:58:15.294569 17070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:15.297881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17070
I20260620 08:58:15.298379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33225
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45763
--webserver_port=46699
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:58:15.299175 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.297432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:36895)
0620 08:58:15.298060 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.298218 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
W20260620 08:58:15.299701 17202 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45763 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45763: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
I20260620 08:58:15.300527 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.300223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:39227)
0620 08:58:15.300365 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.300437 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 08:58:15.301272 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.301034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:43015)
0620 08:58:15.301152 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.301203 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:58:15.301477 17202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:58:15.423621 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:15.423895 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:15.423938 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:15.428129 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:15.428185 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:15.428244 17205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:15.433879 17205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46699
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:15.434970 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:15.436033 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:15.445859 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:15.445868 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:15.446134 17205 server_base.cc:1034] running on GCE node
W20260620 08:58:15.445868 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:15.446605 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:15.447249 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:15.448457 17205 hybrid_clock.cc:648] HybridClock initialized: now 1781945895448433 us; error 32 us; skew 500 ppm
Jun 20 08:58:15 dist-test-slave-fb1v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:15.451435 17205 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:58:15.452353 17205 webserver.cc:460] Webserver started at http://127.0.98.3:46699/ using document root <none> and password file <none>
I20260620 08:58:15.452975 17205 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:15.453038 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:15.456998 17205 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:15.459208 17220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:15.460763 17205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:15.460881 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "b016d74c13c44108be60bc9829b29a09"
format_stamp: "Formatted at 2026-06-20 08:57:33 on dist-test-slave-fb1v"
server_key: "3d1f9e022d5cb06829850867f0745e73"
server_key_iv: "61c610aa7faa9d2c06c273e12c9f2974"
server_key_version: "encryptionkey@0"
I20260620 08:58:15.461439 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:15.473507 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:15.476657 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:15.476842 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:15.477455 17205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:15.478274 17205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:15.478320 17205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:15.478384 17205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:15.478410 17205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:15.486837 17205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33225
I20260620 08:58:15.486877 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33225 every 8 connection(s)
I20260620 08:58:15.487828 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:15.491533 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.489905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:43179)
0620 08:58:15.490354 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.490821 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
W20260620 08:58:15.492056 17334 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45763 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45763: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
I20260620 08:58:15.492601 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.492380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:36201)
0620 08:58:15.492501 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.492548 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 08:58:15.493227 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:15.493035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:58799)
0620 08:58:15.493139 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:15.493182 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:58:15.493433 17334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:58:15.496443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17205
I20260620 08:58:16.142171 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.141698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:40827)
0620 08:58:16.141984 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:16.142050 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260620 08:58:16.302485 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.302055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:54129)
0620 08:58:16.302306 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:16.302370 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260620 08:58:16.494372 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.493951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:52201)
0620 08:58:16.494212 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:16.494275 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260620 08:58:16.536208 17341 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:16.536432 17341 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:16.537575 17341 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
Jun 20 08:58:16 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 08:58:16 dist-test-slave-fb1v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 08:58:16.551851 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.537826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34817 (local address 127.0.98.1:34319)
0620 08:58:16.538233 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:16.538245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:16.538330 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:16.539778 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:16.539784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:16.540331 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:16.540766 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:16.540781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.542184 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.542186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:16.543158 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:16.543167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.543860 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.543863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:16.544062 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:16.544700 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:16.544716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:16.546564 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:16.548694 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:16.548697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:16.548699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:16.549277 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:16.549688 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:16.549692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:16.549694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:16.549936 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:16.550704 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:16.550710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:16.551080 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 08:58:16.551290 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 08:58:16.551689 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":367,"spinlock_wait_cycles":22272,"thread_start_us":72,"threads_started":1}
I20260620 08:58:16.551851 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.537826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33225 (local address 127.0.98.1:34493)
0620 08:58:16.537929 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:16.537944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:16.538045 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:16.539778 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:16.539784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:16.540332 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:16.540768 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:16.540781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.542200 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.542203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:16.543150 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:16.543159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.543866 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.543871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:16.544063 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:16.544704 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:16.544720 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:16.546210 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:16.548590 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:16.548599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:16.548619 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:16.549277 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:16.549688 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:16.549692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:16.549694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:16.549936 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:16.550703 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:16.550710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:16.551078 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 08:58:16.551290 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 08:58:16.551643 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":183,"spinlock_wait_cycles":35584}
I20260620 08:58:16.552374 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.538103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34319 (local address 127.0.98.2:34817)
0620 08:58:16.538385 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:16.538395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:16.538427 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:58:16.539127 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:16.539133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:16.539413 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:16.539710 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:16.540945 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.542046 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.543268 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.543752 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.543894 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:16.546744 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:16.546751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:16.546764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:16.546797 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:16.548576 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:16.549396 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:16.549402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:16.549408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:16.549566 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:16.550056 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:16.550060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:16.550062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:16.550568 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:16.550700 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:16.551429 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 08:58:16.551731 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260620 08:58:16.552644 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:16.537966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34493 (local address 127.0.98.3:33225)
0620 08:58:16.538297 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:16.538321 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:16.538367 (+ 46us) server_negotiation.cc:407] Connection header received
0620 08:58:16.539049 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:16.539055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:16.539389 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:16.539691 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:16.540955 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.542055 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.543261 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:16.543750 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:16.543904 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:16.546436 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:16.546441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:16.546452 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:16.546481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:16.548371 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:16.549396 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:16.549402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:16.549408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:16.549566 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:16.550056 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:16.550068 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:16.550074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:16.550568 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:16.550716 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:16.551429 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 08:58:16.552506 (+ 1077us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
W20260620 08:58:16.554632 16955 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
W20260620 08:58:16.554777 16953 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
I20260620 08:58:16.554847 16953 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765; no voters: 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09
I20260620 08:58:16.555102 17341 raft_consensus.cc:2726] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:58:17.143328 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:17.142936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:58179)
0620 08:58:17.143181 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:17.143234 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260620 08:58:17.303838 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:17.303271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:56631)
0620 08:58:17.303665 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:17.303739 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260620 08:58:17.495710 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:17.495206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:58853)
0620 08:58:17.495486 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:17.495554 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260620 08:58:18.144842 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:18.144222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:41909)
0620 08:58:18.144498 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:18.144555 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260620 08:58:18.304962 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:18.304524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:58845)
0620 08:58:18.304790 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:18.304861 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260620 08:58:18.365655 17350 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:18.365798 17350 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:18.366202 17350 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
W20260620 08:58:18.367023 16953 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
W20260620 08:58:18.367141 16955 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
I20260620 08:58:18.367260 16955 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765; no voters: 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09
I20260620 08:58:18.367408 17350 raft_consensus.cc:2726] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:58:18.496949 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:18.496548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:57267)
0620 08:58:18.496793 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:18.496850 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20260620 08:58:19.146148 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:19.145707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:50143)
0620 08:58:19.145984 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:19.146048 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260620 08:58:19.306102 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:19.305674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:39615)
0620 08:58:19.305934 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:19.305999 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260620 08:58:19.498004 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:19.497625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:58999)
0620 08:58:19.497862 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:19.497915 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260620 08:58:20.147686 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:20.147157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:47885)
0620 08:58:20.147478 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:20.147544 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260620 08:58:20.307325 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:20.306850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:38523)
0620 08:58:20.307160 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:20.307220 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":73,"threads_started":1}
I20260620 08:58:20.499188 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:20.498709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:45551)
0620 08:58:20.499034 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:20.499096 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
I20260620 08:58:21.149029 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:21.148552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:48569)
0620 08:58:21.148870 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:21.148933 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260620 08:58:21.309055 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:21.308306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:32839)
0620 08:58:21.308859 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:21.308938 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":130,"threads_started":1}
I20260620 08:58:21.477191 17360 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:21.477331 17360 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:21.477833 17360 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
W20260620 08:58:21.478639 16953 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
W20260620 08:58:21.478807 16955 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
I20260620 08:58:21.478876 16955 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765; no voters: 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09
I20260620 08:58:21.479002 17360 raft_consensus.cc:2726] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:58:21.500464 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:21.500060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:54265)
0620 08:58:21.500318 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:21.500374 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260620 08:58:22.150214 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:22.149775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:50785)
0620 08:58:22.150062 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:22.150110 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260620 08:58:22.310273 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:22.309821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:34961)
0620 08:58:22.310118 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:22.310178 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260620 08:58:22.501634 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:22.501193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:34727)
0620 08:58:22.501487 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:22.501542 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260620 08:58:23.088423 17365 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:23.088567 17365 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:23.088999 17365 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
W20260620 08:58:23.089823 16955 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
W20260620 08:58:23.089987 16953 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
I20260620 08:58:23.090068 16953 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765; no voters: 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09
I20260620 08:58:23.090209 17365 raft_consensus.cc:2726] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:58:23.151427 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:23.150996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:35071)
0620 08:58:23.151267 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:23.151329 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260620 08:58:23.311522 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:23.311045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:46797)
0620 08:58:23.311342 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:23.311403 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260620 08:58:23.502770 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:23.502354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:58101)
0620 08:58:23.502610 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:23.502668 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260620 08:58:24.152555 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:24.152176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:44493)
0620 08:58:24.152404 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:24.152460 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260620 08:58:24.312711 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:24.312259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:52775)
0620 08:58:24.312559 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:24.312615 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260620 08:58:24.504009 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:24.503481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:57897)
0620 08:58:24.503847 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:24.503913 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":95,"threads_started":1}
I20260620 08:58:25.153832 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:25.153426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:37473)
0620 08:58:25.153678 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:25.153737 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260620 08:58:25.313917 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:25.313436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:42087)
0620 08:58:25.313755 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:25.313816 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260620 08:58:25.505211 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:25.504793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:49233)
0620 08:58:25.505066 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:25.505119 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260620 08:58:26.154999 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:26.154589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:50863)
0620 08:58:26.154839 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:26.154901 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260620 08:58:26.315080 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:26.314653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:36875)
0620 08:58:26.314926 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:26.314985 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260620 08:58:26.506395 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:26.505994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:51453)
0620 08:58:26.506235 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:26.506289 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260620 08:58:27.156400 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:27.155917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:55559)
0620 08:58:27.156228 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:27.156300 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260620 08:58:27.316430 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:27.315934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:60103)
0620 08:58:27.316261 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:27.316332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260620 08:58:27.507668 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:27.507221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:51297)
0620 08:58:27.507475 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:27.507538 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260620 08:58:28.093037 17381 raft_consensus.cc:482] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:28.093165 17381 raft_consensus.cc:504] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b016d74c13c44108be60bc9829b29a09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33225 } } peers { permanent_uuid: "4cdefe194f6c4a1ba4ad6296af8ea0e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34817 } } peers { permanent_uuid: "27dcf0338b164a9aa916defcd8317765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41839 } }
I20260620 08:58:28.093533 17381 leader_election.cc:288] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225), 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817)
W20260620 08:58:28.094310 16955 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cdefe194f6c4a1ba4ad6296af8ea0e9 (127.0.98.2:34817): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
W20260620 08:58:28.094422 16953 leader_election.cc:341] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b016d74c13c44108be60bc9829b29a09 (127.0.98.3:33225): Not found: Tablet not found: bce8039d35a54df9bee325a2e02e623e
I20260620 08:58:28.094508 16953 leader_election.cc:302] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27dcf0338b164a9aa916defcd8317765; no voters: 4cdefe194f6c4a1ba4ad6296af8ea0e9, b016d74c13c44108be60bc9829b29a09
I20260620 08:58:28.094684 17381 raft_consensus.cc:2726] T bce8039d35a54df9bee325a2e02e623e P 27dcf0338b164a9aa916defcd8317765 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:58:28.157725 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:28.157326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:54237)
0620 08:58:28.157570 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:28.157633 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260620 08:58:28.317840 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:28.317398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:37875)
0620 08:58:28.317683 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:28.317745 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260620 08:58:28.508855 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:28.508434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:52031)
0620 08:58:28.508704 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:28.508761 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260620 08:58:29.158998 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:29.158529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:33489)
0620 08:58:29.158836 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:29.158901 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":87,"threads_started":1}
I20260620 08:58:29.319443 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:29.318862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:51027)
0620 08:58:29.319263 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:29.319331 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":147,"threads_started":1}
I20260620 08:58:29.509964 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:29.509576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:35741)
0620 08:58:29.509817 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:29.509873 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260620 08:58:30.160872 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:30.160037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:39997)
0620 08:58:30.160519 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:30.160587 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":72,"threads_started":1}
I20260620 08:58:30.320915 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:30.320435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:44871)
0620 08:58:30.320764 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:30.320822 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260620 08:58:30.499910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 16897
I20260620 08:58:30.506747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17070
I20260620 08:58:30.511282 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:30.510895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:57077)
0620 08:58:30.511140 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:30.511194 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260620 08:58:30.512771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17205
2026-06-20T08:58:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59308 ms)
[----------] 30 tests from SecurityITest (239500 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:30 dist-test-slave-fb1v krb5kdc[17393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:30 dist-test-slave-fb1v krb5kdc[17393](info): set up 2 sockets
Jun 20 08:58:30 dist-test-slave-fb1v krb5kdc[17393](info): commencing operation
krb5kdc: starting...
W20260620 08:58:31.483248 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:31 dist-test-slave-fb1v krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:58:31.607561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46113
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:31.721282 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:31.721545 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:31.721606 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:31.725646 17406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:31.725709 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:31.725735 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:31.725755 17406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:31.725785 17406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:31.730984 17406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:31.732061 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:31.733021 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:31.741997 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:31.741997 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:31.742013 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:31.742312 17406 server_base.cc:1034] running on GCE node
I20260620 08:58:31.742921 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:31.744014 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:31.745167 17406 hybrid_clock.cc:648] HybridClock initialized: now 1781945911745151 us; error 30 us; skew 500 ppm
Jun 20 08:58:31 dist-test-slave-fb1v krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:31.747633 17406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:58:31.748425 17406 webserver.cc:460] Webserver started at http://127.0.98.62:36455/ using document root <none> and password file <none>
I20260620 08:58:31.748944 17406 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:31.749008 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:31.749183 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:31.751130 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "c960fea996af455896f8d528ef15cfde"
format_stamp: "Formatted at 2026-06-20 08:58:31 on dist-test-slave-fb1v"
server_key: "9a523dc1a96662e2ee999e5ede52bd37"
server_key_iv: "b246fd2d8c8875043f1bc6e32b199b45"
server_key_version: "encryptionkey@0"
I20260620 08:58:31.751682 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "c960fea996af455896f8d528ef15cfde"
format_stamp: "Formatted at 2026-06-20 08:58:31 on dist-test-slave-fb1v"
server_key: "9a523dc1a96662e2ee999e5ede52bd37"
server_key_iv: "b246fd2d8c8875043f1bc6e32b199b45"
server_key_version: "encryptionkey@0"
I20260620 08:58:31.755053 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:31.757396 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:31.758622 17406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 08:58:31.758749 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "c960fea996af455896f8d528ef15cfde"
format_stamp: "Formatted at 2026-06-20 08:58:31 on dist-test-slave-fb1v"
server_key: "9a523dc1a96662e2ee999e5ede52bd37"
server_key_iv: "b246fd2d8c8875043f1bc6e32b199b45"
server_key_version: "encryptionkey@0"
I20260620 08:58:31.758872 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:31.798125 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:31.801424 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:31.801602 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:31.808315 17406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46113
I20260620 08:58:31.808566 17473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46113 every 8 connection(s)
I20260620 08:58:31.809252 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:31.812115 17474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:31.814175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17406
I20260620 08:58:31.814296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:31.814658 392 external_mini_cluster.cc:1428] Setting key b07817eb834c48c8c4b3b474f478971d
I20260620 08:58:31.818223 17474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde: Bootstrap starting.
I20260620 08:58:31.820528 17474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:31.821274 17474 log.cc:824] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:58:31 dist-test-slave-fb1v krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:31.823469 17474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde: No bootstrap required, opened a new log
I20260620 08:58:31.827026 17474 raft_consensus.cc:348] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c960fea996af455896f8d528ef15cfde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260620 08:58:31.827268 17474 raft_consensus.cc:374] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:31.827350 17474 raft_consensus.cc:729] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c960fea996af455896f8d528ef15cfde, State: Initialized, Role: FOLLOWER
I20260620 08:58:31.827865 17474 consensus_queue.cc:260] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c960fea996af455896f8d528ef15cfde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260620 08:58:31.828007 17474 raft_consensus.cc:388] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:31.828076 17474 raft_consensus.cc:482] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:31.828176 17474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:31.828471 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:31.816084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.0.98.62:46113)
0620 08:58:31.816531 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:31.816546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:31.816585 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:58:31.817396 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:31.817414 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:31.817733 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:31.818025 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:31.818761 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:31.819773 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:31.820602 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:31.821101 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:31.821245 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:31.824020 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:31.824026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:31.824039 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:31.824066 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:31.825609 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:31.826002 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:31.826007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:31.826011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:31.826114 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:31.826354 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:31.826357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:31.826359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:31.826755 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:31.826954 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:31.827208 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 08:58:31.827657 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260620 08:58:31.829203 17474 raft_consensus.cc:504] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c960fea996af455896f8d528ef15cfde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260620 08:58:31.829527 17474 leader_election.cc:302] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c960fea996af455896f8d528ef15cfde; no voters:
I20260620 08:58:31.829836 17474 leader_election.cc:288] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:31.829933 17479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:31.830104 17479 raft_consensus.cc:686] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [term 1 LEADER]: Becoming Leader. State: Replica: c960fea996af455896f8d528ef15cfde, State: Running, Role: LEADER
I20260620 08:58:31.830478 17479 consensus_queue.cc:237] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c960fea996af455896f8d528ef15cfde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } }
I20260620 08:58:31.831167 17474 sys_catalog.cc:564] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:31.831794 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [sys.catalog]: SysCatalogTable state changed. Reason: New leader c960fea996af455896f8d528ef15cfde. Latest consensus state: current_term: 1 leader_uuid: "c960fea996af455896f8d528ef15cfde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c960fea996af455896f8d528ef15cfde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } }
I20260620 08:58:31.832170 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:31.832247 17480 sys_catalog.cc:455] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c960fea996af455896f8d528ef15cfde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c960fea996af455896f8d528ef15cfde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } }
I20260620 08:58:31.832466 17480 sys_catalog.cc:458] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:31.832639 17488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:31.835599 17488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:31.841480 17488 catalog_manager.cc:1347] Generated new cluster ID: 8b42de5f2f6e4a178bfd094df3a28233
I20260620 08:58:31.841548 17488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:31.848121 17488 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:31.849303 17488 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:31.855919 17488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde: Generated new TSK 0
I20260620 08:58:31.856719 17488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:58:31.888675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:31.996480 17500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:31.996737 17500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:31.996778 17500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:32.000802 17500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:32.000859 17500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:32.000929 17500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:32.006104 17500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:32.007148 17500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:32.008139 17500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:32.018231 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.018225 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.018225 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:32.018545 17500 server_base.cc:1034] running on GCE node
I20260620 08:58:32.018925 17500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:32.019594 17500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:32.020790 17500 hybrid_clock.cc:648] HybridClock initialized: now 1781945912020755 us; error 49 us; skew 500 ppm
Jun 20 08:58:32 dist-test-slave-fb1v krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:32.026131 17500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:58:32.027047 17500 webserver.cc:460] Webserver started at http://127.0.98.1:44835/ using document root <none> and password file <none>
I20260620 08:58:32.027566 17500 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:32.027653 17500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:32.027848 17500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:32.029784 17500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "63061bb8a18f846a723d7aa38ce967db"
server_key_iv: "23b8ec7e791f56c919548ea004681194"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.030290 17500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "63061bb8a18f846a723d7aa38ce967db"
server_key_iv: "23b8ec7e791f56c919548ea004681194"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.034104 17500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 08:58:32.036454 17515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.037601 17500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 08:58:32.037717 17500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "63061bb8a18f846a723d7aa38ce967db"
server_key_iv: "23b8ec7e791f56c919548ea004681194"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.037809 17500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:32.056257 17500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:32.059195 17500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:32.059376 17500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:32.059939 17500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:32.060714 17500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:32.060758 17500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.060810 17500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:32.060838 17500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.069609 17500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45427
I20260620 08:58:32.069674 17628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45427 every 8 connection(s)
I20260620 08:58:32.070497 17500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:32.074898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17500
I20260620 08:58:32.075009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:58:32.075263 392 external_mini_cluster.cc:1428] Setting key 492c31928ba5ae4058175089a6c34df1
Jun 20 08:58:32 dist-test-slave-fb1v krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:58:32.084034 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.072276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42591 (local address 127.0.98.62:46113)
0620 08:58:32.072432 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.072437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.073297 (+ 860us) server_negotiation.cc:407] Connection header received
0620 08:58:32.074343 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.074346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.074397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.074477 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:32.075654 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.076374 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.077511 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.077828 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.077936 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.080787 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:32.080790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:32.080792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:32.080814 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:32.082141 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.082603 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.082607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.082608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.082683 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.082974 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.082978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.082979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.083150 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:32.083232 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.083736 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.083858 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:58:32.084836 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.072659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.98.1:42591)
0620 08:58:32.073128 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.073176 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.074114 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.074619 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.074626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.075000 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:32.075505 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.075520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.076504 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.076507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.077376 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.077387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.078013 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.078018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.078238 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.078893 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:32.078917 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:32.080628 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:32.082247 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:32.082253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:32.082266 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:32.082515 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:32.082761 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:32.082764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:32.082766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:32.082893 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:32.083244 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:32.083250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:32.083489 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.083696 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.084005 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260620 08:58:32.085881 17629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46113
I20260620 08:58:32.086112 17629 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:58:32.086591 17629 heartbeater.cc:511] Master 127.0.98.62:46113 requested a full tablet report, sending...
I20260620 08:58:32.088189 17438 ts_manager.cc:194] Registered new tserver with Master: 1a13ae4e6d8a4648ac5492cc9d4e20c9 (127.0.98.1:45427)
I20260620 08:58:32.089346 17438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:58:32.100421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:32.209769 17634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:32.210034 17634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:32.210093 17634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:32.214169 17634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:32.214226 17634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:32.214305 17634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:32.219839 17634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:32.220865 17634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:32.222054 17634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:32.231256 17642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.231283 17640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.231287 17639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:32.231416 17634 server_base.cc:1034] running on GCE node
I20260620 08:58:32.231963 17634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:32.232637 17634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:32.233821 17634 hybrid_clock.cc:648] HybridClock initialized: now 1781945912233801 us; error 30 us; skew 500 ppm
Jun 20 08:58:32 dist-test-slave-fb1v krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:32.236688 17634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:58:32.237524 17634 webserver.cc:460] Webserver started at http://127.0.98.2:33579/ using document root <none> and password file <none>
I20260620 08:58:32.238023 17634 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:32.238083 17634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:32.238256 17634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:32.240247 17634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "61527f4bff984dfabd8a878976c5a5b9"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "4f4748cdc8e5f324d32b940700d22efa"
server_key_iv: "10c2203b4ca7dc7e96b56451165cb8d5"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.240753 17634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61527f4bff984dfabd8a878976c5a5b9"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "4f4748cdc8e5f324d32b940700d22efa"
server_key_iv: "10c2203b4ca7dc7e96b56451165cb8d5"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.244418 17634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:32.246667 17649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.247881 17634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:32.247980 17634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "61527f4bff984dfabd8a878976c5a5b9"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "4f4748cdc8e5f324d32b940700d22efa"
server_key_iv: "10c2203b4ca7dc7e96b56451165cb8d5"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.248068 17634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:32.265161 17634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:32.268201 17634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:32.268352 17634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:32.268859 17634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:32.269619 17634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:32.269665 17634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.269702 17634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:32.269716 17634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.278052 17634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33905
I20260620 08:58:32.278080 17762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33905 every 8 connection(s)
I20260620 08:58:32.278872 17634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:32.286307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17634
I20260620 08:58:32.286425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:58:32.286703 392 external_mini_cluster.cc:1428] Setting key 656d62e7e2cfd90ef901be2d2af804d0
Jun 20 08:58:32 dist-test-slave-fb1v krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:32.291535 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.280556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54931 (local address 127.0.98.62:46113)
0620 08:58:32.280676 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.280680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.281466 (+ 786us) server_negotiation.cc:407] Connection header received
0620 08:58:32.282316 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.282319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.282374 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.282448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:32.283538 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.284118 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.285231 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.285459 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.285545 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.288102 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:32.288106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:32.288109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:32.288135 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:32.289490 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.290062 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.290065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.290067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.290157 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.290431 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.290434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.290435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.290649 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:32.290754 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.291256 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.291381 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:58:32.292347 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.280904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.98.2:54931)
0620 08:58:32.281330 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.281369 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.282128 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.282625 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.282632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.282960 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:32.283397 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.283409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.284236 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.284239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.285115 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.285125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.285591 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.285595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.285752 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.286318 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:32.286335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:32.287936 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:32.289604 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:32.289611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:32.289626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:32.289970 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:32.290239 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:32.290241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:32.290242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:32.290364 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:32.290778 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:32.290783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:32.291023 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.291187 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.291449 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260620 08:58:32.293501 17763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46113
I20260620 08:58:32.293728 17763 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:32.294198 17763 heartbeater.cc:511] Master 127.0.98.62:46113 requested a full tablet report, sending...
I20260620 08:58:32.295220 17438 ts_manager.cc:194] Registered new tserver with Master: 61527f4bff984dfabd8a878976c5a5b9 (127.0.98.2:33905)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:58:32.296013 17438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:58:32.312166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:32.418167 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:32.418408 17768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:32.418553 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:32.422603 17768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:32.422664 17768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:32.422741 17768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:32.428089 17768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:32.429045 17768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:32.429968 17768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:32.439188 17773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:32.439247 17768 server_base.cc:1034] running on GCE node
W20260620 08:58:32.439200 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.439234 17774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:32.439894 17768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:32.440475 17768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:32.441650 17768 hybrid_clock.cc:648] HybridClock initialized: now 1781945912441627 us; error 32 us; skew 500 ppm
Jun 20 08:58:32 dist-test-slave-fb1v krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:32.444270 17768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:58:32.445050 17768 webserver.cc:460] Webserver started at http://127.0.98.3:36141/ using document root <none> and password file <none>
I20260620 08:58:32.445545 17768 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:32.445601 17768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:32.445762 17768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:32.447732 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "8835551fa5f446bea84771e9b368d43b"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "39d07e84dbfe151b458218f975ce852f"
server_key_iv: "1e9620b6e5f3d618df26caf1d8af9641"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.448210 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8835551fa5f446bea84771e9b368d43b"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "39d07e84dbfe151b458218f975ce852f"
server_key_iv: "1e9620b6e5f3d618df26caf1d8af9641"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.451548 17768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 08:58:32.453966 17783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.455096 17768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 08:58:32.455183 17768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "8835551fa5f446bea84771e9b368d43b"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "39d07e84dbfe151b458218f975ce852f"
server_key_iv: "1e9620b6e5f3d618df26caf1d8af9641"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.455262 17768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:32.469039 17768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:32.472247 17768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:32.472424 17768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:32.472954 17768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:32.473753 17768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:32.473812 17768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.473855 17768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:32.473881 17768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.482616 17768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46151
I20260620 08:58:32.482656 17896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46151 every 8 connection(s)
I20260620 08:58:32.483444 17768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:32.487886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17768
I20260620 08:58:32.488015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:58:32.488319 392 external_mini_cluster.cc:1428] Setting key 13fa54aef1d43f316fa832d35fe4af05
Jun 20 08:58:32 dist-test-slave-fb1v krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:32.496670 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.485229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56775 (local address 127.0.98.62:46113)
0620 08:58:32.485383 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.485386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.486385 (+ 999us) server_negotiation.cc:407] Connection header received
0620 08:58:32.487277 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.487281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.487340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.487426 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:32.488877 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.489588 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.490840 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.491058 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.491138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.493606 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:32.493609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:32.493611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:32.493635 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:32.494856 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.495343 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.495345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.495346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.495411 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.495749 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.495752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.495753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.495941 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:32.496022 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.496452 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.496551 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:58:32.497426 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.485552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.98.3:56775)
0620 08:58:32.486208 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.486249 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.487071 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.487626 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.487638 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.488108 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:32.488728 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.488747 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.489713 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.489716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.490728 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.490738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.491171 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.491174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.491319 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.491883 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:32.491900 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:32.493378 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:32.494967 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:32.494972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:32.494984 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:32.495260 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:32.495488 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:32.495490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:32.495491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:32.495663 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:32.496027 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:32.496031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:32.496254 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.496418 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.496653 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":110,"threads_started":1}
I20260620 08:58:32.498406 17897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46113
I20260620 08:58:32.498636 17897 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:32.499086 17897 heartbeater.cc:511] Master 127.0.98.62:46113 requested a full tablet report, sending...
I20260620 08:58:32.500089 17438 ts_manager.cc:194] Registered new tserver with Master: 8835551fa5f446bea84771e9b368d43b (127.0.98.3:46151)
I20260620 08:58:32.500615 17438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56775
I20260620 08:58:32.501884 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:58:32.510466 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.503886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.0.98.62:46113)
0620 08:58:32.503998 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.504003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.504169 (+ 166us) server_negotiation.cc:407] Connection header received
0620 08:58:32.504279 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.504281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.504323 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.504384 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:32.505037 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.505534 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.506319 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.506532 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.506615 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.507889 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:32.507892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:32.507894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:32.507915 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:32.508989 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.509348 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.509351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.509353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.509430 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:32.509662 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:32.509666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:32.509667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:32.509876 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:32.509991 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.510209 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.510318 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:58:32.513387 17437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:58:32.516245 17437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:58:32.530107 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.524859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.98.3:46151)
0620 08:58:32.525539 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.525553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.525583 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:58:32.525644 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.525647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.525815 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.525951 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:32.526113 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.527241 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.528762 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.529367 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.529468 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.529747 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.529835 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.529933 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":158,"threads_started":1}
I20260620 08:58:32.530103 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.524216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46151 (local address 127.0.0.1:54676)
0620 08:58:32.524795 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.524808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.524892 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.525928 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.525931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.525945 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:32.526024 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.526030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.527370 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.527373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.528634 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.528644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.529482 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.529485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.529626 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.529689 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.529807 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.529976 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":134,"thread_start_us":97,"threads_started":1}
I20260620 08:58:32.530115 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.524088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45427 (local address 127.0.0.1:58766)
0620 08:58:32.524691 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.524737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.524870 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.525740 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.525747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.525770 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:32.525918 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.525925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.527408 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.527410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.528744 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.528755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.529515 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.529519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.529637 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.529691 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.529795 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.529959 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":163,"spinlock_wait_cycles":2816,"thread_start_us":79,"threads_started":1}
I20260620 08:58:32.530737 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.524234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.0.98.1:45427)
0620 08:58:32.525267 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.525279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.525300 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:58:32.525388 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.525393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.525587 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.525742 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:32.526057 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.527244 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.528879 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.529404 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.529502 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.529647 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.530467 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.530591 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":105,"threads_started":1}
I20260620 08:58:32.531325 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.524435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33905 (local address 127.0.0.1:42496)
0620 08:58:32.525097 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.525109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.525183 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.525774 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.525777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.525795 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:32.525927 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.525934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.527159 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.527162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.528765 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.528772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.530949 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.530953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.531052 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.531076 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.531120 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.531186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":210,"thread_start_us":94,"threads_started":1}
I20260620 08:58:32.532028 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.525107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.2:33905)
0620 08:58:32.525408 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.525417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.525435 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:58:32.525503 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.525507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.525644 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.526063 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:32.526200 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.527013 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.529708 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.530803 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.531572 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.531734 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.531786 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.531884 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
I20260620 08:58:32.533818 17563 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad757e079e7420eb18368e14233a91a (DEFAULT_TABLE table=test-table [id=916ec20fb7b84523aa74126007954c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:32.533852 17831 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad757e079e7420eb18368e14233a91a (DEFAULT_TABLE table=test-table [id=916ec20fb7b84523aa74126007954c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:32.534853 17563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad757e079e7420eb18368e14233a91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:32.534860 17831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad757e079e7420eb18368e14233a91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:32.534575 17697 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad757e079e7420eb18368e14233a91a (DEFAULT_TABLE table=test-table [id=916ec20fb7b84523aa74126007954c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:32.535459 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad757e079e7420eb18368e14233a91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:32.540954 17914 tablet_bootstrap.cc:492] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: Bootstrap starting.
I20260620 08:58:32.542599 17916 tablet_bootstrap.cc:492] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Bootstrap starting.
I20260620 08:58:32.543121 17914 tablet_bootstrap.cc:654] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:32.543937 17915 tablet_bootstrap.cc:492] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Bootstrap starting.
I20260620 08:58:32.544143 17914 log.cc:824] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:32.544423 17916 tablet_bootstrap.cc:654] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:32.545584 17916 log.cc:824] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:32.545945 17915 tablet_bootstrap.cc:654] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:32.545989 17914 tablet_bootstrap.cc:492] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: No bootstrap required, opened a new log
I20260620 08:58:32.546218 17914 ts_tablet_manager.cc:1397] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260620 08:58:32.547067 17915 log.cc:824] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:32.547520 17916 tablet_bootstrap.cc:492] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: No bootstrap required, opened a new log
I20260620 08:58:32.547719 17916 ts_tablet_manager.cc:1397] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:58:32.548945 17915 tablet_bootstrap.cc:492] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: No bootstrap required, opened a new log
I20260620 08:58:32.549110 17915 ts_tablet_manager.cc:1397] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:58:32.549188 17914 raft_consensus.cc:348] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.549440 17914 raft_consensus.cc:374] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:32.549520 17914 raft_consensus.cc:729] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a13ae4e6d8a4648ac5492cc9d4e20c9, State: Initialized, Role: FOLLOWER
I20260620 08:58:32.549937 17914 consensus_queue.cc:260] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.550532 17629 heartbeater.cc:503] Master 127.0.98.62:46113 was elected leader, sending a full tablet report...
I20260620 08:58:32.550784 17914 ts_tablet_manager.cc:1428] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:58:32.551831 17916 raft_consensus.cc:348] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.552214 17916 raft_consensus.cc:374] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:32.552380 17916 raft_consensus.cc:729] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61527f4bff984dfabd8a878976c5a5b9, State: Initialized, Role: FOLLOWER
I20260620 08:58:32.552565 17915 raft_consensus.cc:348] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.552806 17915 raft_consensus.cc:374] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:32.552850 17915 raft_consensus.cc:729] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8835551fa5f446bea84771e9b368d43b, State: Initialized, Role: FOLLOWER
I20260620 08:58:32.552904 17916 consensus_queue.cc:260] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.553326 17915 consensus_queue.cc:260] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.553514 17763 heartbeater.cc:503] Master 127.0.98.62:46113 was elected leader, sending a full tablet report...
I20260620 08:58:32.554046 17897 heartbeater.cc:503] Master 127.0.98.62:46113 was elected leader, sending a full tablet report...
I20260620 08:58:32.554277 17915 ts_tablet_manager.cc:1428] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:58:32.554925 17916 ts_tablet_manager.cc:1428] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260620 08:58:32.571735 17630 tablet.cc:2369] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:32.649744 17921 raft_consensus.cc:482] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:32.649988 17921 raft_consensus.cc:504] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.651105 17921 leader_election.cc:288] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a13ae4e6d8a4648ac5492cc9d4e20c9 (127.0.98.1:45427), 8835551fa5f446bea84771e9b368d43b (127.0.98.3:46151)
I20260620 08:58:32.655540 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.651306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46151 (local address 127.0.98.2:39211)
0620 08:58:32.651434 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.651449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.651551 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.651855 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.651858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.651877 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:32.651998 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.652004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.653047 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.653050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.654273 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.654285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.655123 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.655126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.655214 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.655234 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.655315 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.655392 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:58:32.655539 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.651425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39211 (local address 127.0.98.3:46151)
0620 08:58:32.651567 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.651572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.651636 (+ 64us) server_negotiation.cc:407] Connection header received
0620 08:58:32.651703 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.651705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.651757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.651845 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:32.652116 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.652811 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.654424 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.654975 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.655095 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.655159 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.655344 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.655418 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:58:32.655706 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.651306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45427 (local address 127.0.98.2:47125)
0620 08:58:32.651855 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.651867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:32.651978 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:32.652319 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:32.652323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:32.652345 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:32.652445 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.652450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.653355 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.653358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.654552 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:32.654560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.655332 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.655335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:32.655425 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.655446 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:58:32.655514 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:58:32.655570 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":230,"thread_start_us":156,"threads_started":1}
I20260620 08:58:32.655761 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.651484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47125 (local address 127.0.98.1:45427)
0620 08:58:32.651646 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.651651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.651952 (+ 301us) server_negotiation.cc:407] Connection header received
0620 08:58:32.652112 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.652116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.652167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.652263 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:32.652554 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.653241 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.654675 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.655217 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.655295 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.655354 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.655540 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.655653 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:58:32.656323 17583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad757e079e7420eb18368e14233a91a" candidate_uuid: "61527f4bff984dfabd8a878976c5a5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" is_pre_election: true
I20260620 08:58:32.656368 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad757e079e7420eb18368e14233a91a" candidate_uuid: "61527f4bff984dfabd8a878976c5a5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8835551fa5f446bea84771e9b368d43b" is_pre_election: true
I20260620 08:58:32.656612 17583 raft_consensus.cc:2454] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61527f4bff984dfabd8a878976c5a5b9 in term 0.
I20260620 08:58:32.656623 17851 raft_consensus.cc:2454] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61527f4bff984dfabd8a878976c5a5b9 in term 0.
I20260620 08:58:32.657074 17653 leader_election.cc:302] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61527f4bff984dfabd8a878976c5a5b9, 8835551fa5f446bea84771e9b368d43b; no voters:
I20260620 08:58:32.657332 17921 raft_consensus.cc:2781] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:58:32.657418 17921 raft_consensus.cc:482] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:58:32.657466 17921 raft_consensus.cc:3037] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:32.658524 17921 raft_consensus.cc:504] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.658861 17921 leader_election.cc:288] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a13ae4e6d8a4648ac5492cc9d4e20c9 (127.0.98.1:45427), 8835551fa5f446bea84771e9b368d43b (127.0.98.3:46151)
I20260620 08:58:32.659194 17583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad757e079e7420eb18368e14233a91a" candidate_uuid: "61527f4bff984dfabd8a878976c5a5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9"
I20260620 08:58:32.659222 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad757e079e7420eb18368e14233a91a" candidate_uuid: "61527f4bff984dfabd8a878976c5a5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8835551fa5f446bea84771e9b368d43b"
I20260620 08:58:32.659297 17583 raft_consensus.cc:3037] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:32.659332 17851 raft_consensus.cc:3037] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:32.660311 17583 raft_consensus.cc:2454] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61527f4bff984dfabd8a878976c5a5b9 in term 1.
I20260620 08:58:32.660503 17851 raft_consensus.cc:2454] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61527f4bff984dfabd8a878976c5a5b9 in term 1.
I20260620 08:58:32.660585 17650 leader_election.cc:302] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a13ae4e6d8a4648ac5492cc9d4e20c9, 61527f4bff984dfabd8a878976c5a5b9; no voters:
I20260620 08:58:32.660744 17921 raft_consensus.cc:2781] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:32.660950 17921 raft_consensus.cc:686] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 61527f4bff984dfabd8a878976c5a5b9, State: Running, Role: LEADER
I20260620 08:58:32.661293 17921 consensus_queue.cc:237] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } }
I20260620 08:58:32.664458 17436 catalog_manager.cc:5495] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61527f4bff984dfabd8a878976c5a5b9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "61527f4bff984dfabd8a878976c5a5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61527f4bff984dfabd8a878976c5a5b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260620 08:58:32.682994 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.679476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.0.98.2:33905)
0620 08:58:32.679647 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.679652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.679668 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:58:32.679778 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.679780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.679835 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.679924 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:58:32.680154 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.680683 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.681531 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.681756 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.681839 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.682113 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:58:32.682536 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:58:32.682624 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.682770 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.682831 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:58:32.688514 17851 raft_consensus.cc:1261] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 1 FOLLOWER]: Refusing update from remote peer 61527f4bff984dfabd8a878976c5a5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:32.688514 17583 raft_consensus.cc:1261] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 1 FOLLOWER]: Refusing update from remote peer 61527f4bff984dfabd8a878976c5a5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:32.689170 17924 consensus_queue.cc:1035] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a13ae4e6d8a4648ac5492cc9d4e20c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:32.689339 17921 consensus_queue.cc:1035] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8835551fa5f446bea84771e9b368d43b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:32.705544 17436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:32.705729 17436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:32.708128 17436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c960fea996af455896f8d528ef15cfde: Sending DeleteTablet for 3 replicas of tablet 2ad757e079e7420eb18368e14233a91a
I20260620 08:58:32.708976 17831 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad757e079e7420eb18368e14233a91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54676
I20260620 08:58:32.708966 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad757e079e7420eb18368e14233a91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42496
I20260620 08:58:32.709456 17938 tablet_replica.cc:330] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: stopping tablet replica
I20260620 08:58:32.709435 17563 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad757e079e7420eb18368e14233a91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260620 08:58:32.709792 17938 raft_consensus.cc:2229] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:32.709846 17939 tablet_replica.cc:330] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9: stopping tablet replica
I20260620 08:58:32.710121 17938 raft_consensus.cc:2258] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:32.710253 17939 raft_consensus.cc:2229] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:32.710467 17937 tablet_replica.cc:330] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: stopping tablet replica
I20260620 08:58:32.710578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17500
I20260620 08:58:32.710567 17939 raft_consensus.cc:2258] T 2ad757e079e7420eb18368e14233a91a P 1a13ae4e6d8a4648ac5492cc9d4e20c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:32.710822 17937 raft_consensus.cc:2229] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:58:32.711205 17937 raft_consensus.cc:2258] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:32.712570 17937 ts_tablet_manager.cc:1905] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:32.713543 17938 ts_tablet_manager.cc:1905] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:32.715178 17937 ts_tablet_manager.cc:1918] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:32.715276 17937 log.cc:1196] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/2ad757e079e7420eb18368e14233a91a
I20260620 08:58:32.715648 17937 ts_tablet_manager.cc:1939] T 2ad757e079e7420eb18368e14233a91a P 61527f4bff984dfabd8a878976c5a5b9: Deleting consensus metadata
I20260620 08:58:32.716097 17938 ts_tablet_manager.cc:1918] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:32.716241 17938 log.cc:1196] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/2ad757e079e7420eb18368e14233a91a
W20260620 08:58:32.716390 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45427 (error 108)
I20260620 08:58:32.716580 17938 ts_tablet_manager.cc:1939] T 2ad757e079e7420eb18368e14233a91a P 8835551fa5f446bea84771e9b368d43b: Deleting consensus metadata
I20260620 08:58:32.716881 17423 catalog_manager.cc:4841] TS 61527f4bff984dfabd8a878976c5a5b9 (127.0.98.2:33905): tablet 2ad757e079e7420eb18368e14233a91a (table test-table [id=916ec20fb7b84523aa74126007954c99]) successfully deleted
I20260620 08:58:32.717262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17634
I20260620 08:58:32.717882 17425 catalog_manager.cc:4841] TS 8835551fa5f446bea84771e9b368d43b (127.0.98.3:46151): tablet 2ad757e079e7420eb18368e14233a91a (table test-table [id=916ec20fb7b84523aa74126007954c99]) successfully deleted
I20260620 08:58:32.718746 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.718463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45427 (local address 127.0.0.1:58780)
0620 08:58:32.718594 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:32.718664 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:58:32.718915 17422 catalog_manager.cc:4584] TS 1a13ae4e6d8a4648ac5492cc9d4e20c9 (127.0.98.1:45427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ad757e079e7420eb18368e14233a91a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45427: connect: Connection refused (error 111)
I20260620 08:58:32.722545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17768
I20260620 08:58:32.726846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17406
2026-06-20T08:58:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T08:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:32Z Disabled control of system clock
I20260620 08:58:32.754129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33665
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:58:32.858390 17944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:32.858642 17944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:32.858675 17944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:32.862610 17944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:32.862679 17944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:32.862694 17944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:32.862711 17944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:32.862725 17944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:32.868255 17944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:32.869294 17944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:32.870563 17944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:32.878307 17950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.878329 17949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:32.878355 17952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:32.878530 17944 server_base.cc:1034] running on GCE node
I20260620 08:58:32.879094 17944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:32.880124 17944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:32.881265 17944 hybrid_clock.cc:648] HybridClock initialized: now 1781945912881249 us; error 30 us; skew 500 ppm
I20260620 08:58:32.882905 17944 webserver.cc:460] Webserver started at http://127.0.98.62:42809/ using document root <none> and password file <none>
I20260620 08:58:32.883423 17944 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:32.883486 17944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:32.883695 17944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:32.885572 17944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "fdd8d798c5f744619bcd77fff48cc3ce"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "4141688f661279da9a00683dc15d91b2"
server_key_iv: "4faa74d031dba622bbcda343b4b39d40"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.886093 17944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdd8d798c5f744619bcd77fff48cc3ce"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "4141688f661279da9a00683dc15d91b2"
server_key_iv: "4faa74d031dba622bbcda343b4b39d40"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.889449 17944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:32.891578 17958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:32.892704 17944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:32.892802 17944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "fdd8d798c5f744619bcd77fff48cc3ce"
format_stamp: "Formatted at 2026-06-20 08:58:32 on dist-test-slave-fb1v"
server_key: "4141688f661279da9a00683dc15d91b2"
server_key_iv: "4faa74d031dba622bbcda343b4b39d40"
server_key_version: "encryptionkey@0"
I20260620 08:58:32.892899 17944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:32.906020 17944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:32.906615 17944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:32.906769 17944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:32.913340 17944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33665
I20260620 08:58:32.913358 18010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33665 every 8 connection(s)
I20260620 08:58:32.914212 17944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:32.917207 18011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:32.919186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 17944
I20260620 08:58:32.919327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:32.919628 392 external_mini_cluster.cc:1428] Setting key 6b6b42a54c3853f0b02a4217eb77bb98
I20260620 08:58:32.923146 18011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce: Bootstrap starting.
I20260620 08:58:32.925542 18011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:32.926312 18011 log.cc:824] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:32.928135 18011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce: No bootstrap required, opened a new log
I20260620 08:58:32.928380 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:32.920910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.0.98.62:33665)
0620 08:58:32.921482 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:32.921496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:32.921539 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:58:32.922181 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:32.922196 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:32.922570 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:32.922834 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:32.923156 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.924177 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.925305 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:32.925801 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:32.925920 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:32.926726 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:32.926732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:32.926744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:32.927115 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:32.927169 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:32.927188 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:32.927272 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:32.927404 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 08:58:32.927678 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260620 08:58:32.932212 18011 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260620 08:58:32.932436 18011 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:32.932513 18011 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd8d798c5f744619bcd77fff48cc3ce, State: Initialized, Role: FOLLOWER
I20260620 08:58:32.932901 18011 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260620 08:58:32.933027 18011 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:32.933090 18011 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:32.933185 18011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:32.934092 18011 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260620 08:58:32.934417 18011 leader_election.cc:302] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd8d798c5f744619bcd77fff48cc3ce; no voters:
I20260620 08:58:32.934707 18011 leader_election.cc:288] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:32.934831 18016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:32.935055 18016 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [term 1 LEADER]: Becoming Leader. State: Replica: fdd8d798c5f744619bcd77fff48cc3ce, State: Running, Role: LEADER
I20260620 08:58:32.935349 18016 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260620 08:58:32.935878 18011 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:32.937101 18017 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } } }
I20260620 08:58:32.937126 18018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd8d798c5f744619bcd77fff48cc3ce. Latest consensus state: current_term: 1 leader_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd8d798c5f744619bcd77fff48cc3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } } }
I20260620 08:58:32.937218 18017 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:32.937224 18018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:32.938124 18026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:32.940644 18026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:32.946321 18026 catalog_manager.cc:1347] Generated new cluster ID: 31fc79b6b68c48a4a5768d570575bf78
I20260620 08:58:32.946444 18026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:32.959580 18026 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:32.960606 18026 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:32.966857 18026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce: Generated new TSK 0
I20260620 08:58:32.967482 18026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:32.975087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33665
--builtin_ntp_servers=127.0.98.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:58:33.085989 18035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:33.086257 18035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:33.086300 18035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:33.090365 18035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:33.090432 18035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:33.090493 18035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:33.095361 18035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:33.096511 18035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:33.097872 18035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:33.107089 18040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:33.107147 18043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:33.107076 18041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:33.107501 18035 server_base.cc:1034] running on GCE node
I20260620 08:58:33.107928 18035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:33.108570 18035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:33.109726 18035 hybrid_clock.cc:648] HybridClock initialized: now 1781945913109707 us; error 31 us; skew 500 ppm
I20260620 08:58:33.111672 18035 webserver.cc:460] Webserver started at http://127.0.98.1:33413/ using document root <none> and password file <none>
I20260620 08:58:33.112378 18035 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:33.112457 18035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:33.112658 18035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:33.114657 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "08493386270e4d8eac7d9c4b7372e048"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "eaede52b40f0117e3cbe67efcb236b2c"
server_key_iv: "d9d54f1c23bfa9bc51137a2cf6ed7887"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.115172 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08493386270e4d8eac7d9c4b7372e048"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "eaede52b40f0117e3cbe67efcb236b2c"
server_key_iv: "d9d54f1c23bfa9bc51137a2cf6ed7887"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.119098 18035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 08:58:33.121526 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.122720 18035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:33.122823 18035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "08493386270e4d8eac7d9c4b7372e048"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "eaede52b40f0117e3cbe67efcb236b2c"
server_key_iv: "d9d54f1c23bfa9bc51137a2cf6ed7887"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.122922 18035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:33.144634 18035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:33.145309 18035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:33.145478 18035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:33.146044 18035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:33.146960 18035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:33.147006 18035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.147058 18035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:33.147086 18035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.155761 18035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46583
I20260620 08:58:33.155786 18162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46583 every 8 connection(s)
I20260620 08:58:33.156625 18035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:33.161247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18035
I20260620 08:58:33.161365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:58:33.161633 392 external_mini_cluster.cc:1428] Setting key c0c7cf016ada3b5416944dc5e1094106
I20260620 08:58:33.163826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33665
--builtin_ntp_servers=127.0.98.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:58:33.167268 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.158431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57151 (local address 127.0.98.62:33665)
0620 08:58:33.158585 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.158589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.159668 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 08:58:33.161001 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.161013 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.161070 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.161166 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:33.162355 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.162937 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.164816 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.165165 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.165291 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.166788 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:33.166791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:33.166793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:33.166805 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:33.166825 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:33.166833 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:33.166892 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.167062 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.167130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:58:33.168241 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.158829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33665 (local address 127.0.98.1:57151)
0620 08:58:33.159481 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.159521 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.160801 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.161349 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.161358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.161462 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:33.162144 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.162166 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.163188 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.163194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.164617 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.164637 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.165381 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.165388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.165659 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.166496 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:33.166498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:33.166540 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:33.166610 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:33.166904 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:33.166912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:33.166940 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.167052 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.167357 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260620 08:58:33.169447 18163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33665
I20260620 08:58:33.169689 18163 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:33.170202 18163 heartbeater.cc:511] Master 127.0.98.62:33665 requested a full tablet report, sending...
I20260620 08:58:33.171810 17974 ts_manager.cc:194] Registered new tserver with Master: 08493386270e4d8eac7d9c4b7372e048 (127.0.98.1:46583)
I20260620 08:58:33.173502 17974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57151
W20260620 08:58:33.272643 18166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:33.272894 18166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:33.272935 18166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:33.277215 18166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:33.277263 18166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:33.277318 18166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:33.282042 18166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:33.283022 18166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:33.284348 18166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:33.293610 18174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:33.293632 18171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:33.293792 18166 server_base.cc:1034] running on GCE node
W20260620 08:58:33.293610 18172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:33.294232 18166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:33.294852 18166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:33.296021 18166 hybrid_clock.cc:648] HybridClock initialized: now 1781945913296005 us; error 32 us; skew 500 ppm
I20260620 08:58:33.297839 18166 webserver.cc:460] Webserver started at http://127.0.98.2:35029/ using document root <none> and password file <none>
I20260620 08:58:33.298506 18166 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:33.298576 18166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:33.298741 18166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:33.300808 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "5edd0a478057497b9f35de9c488d2c75"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "829bd4191f12da15581859cc87a65247"
server_key_iv: "816d995211bca5c28abceb73b8d3fb74"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.301331 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5edd0a478057497b9f35de9c488d2c75"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "829bd4191f12da15581859cc87a65247"
server_key_iv: "816d995211bca5c28abceb73b8d3fb74"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.304827 18166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:33.307232 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.308485 18166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:33.308672 18166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "5edd0a478057497b9f35de9c488d2c75"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "829bd4191f12da15581859cc87a65247"
server_key_iv: "816d995211bca5c28abceb73b8d3fb74"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.308796 18166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:33.335959 18166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:33.336702 18166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:33.336880 18166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:33.337492 18166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:33.338425 18166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:33.338472 18166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.338524 18166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:33.338552 18166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.347157 18166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32791
I20260620 08:58:33.347244 18293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32791 every 8 connection(s)
I20260620 08:58:33.348248 18166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:33.351770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18166
I20260620 08:58:33.351886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:58:33.352195 392 external_mini_cluster.cc:1428] Setting key a8b1fe333538f03f723273e6ad8c786d
I20260620 08:58:33.354473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33665
--builtin_ntp_servers=127.0.98.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:58:33.358692 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.350216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56157 (local address 127.0.98.62:33665)
0620 08:58:33.350375 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.350380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.351529 (+ 1149us) server_negotiation.cc:407] Connection header received
0620 08:58:33.353303 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.353307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.353349 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.353431 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:33.354531 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.355250 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.356869 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.357087 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.357172 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.358276 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:33.358279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:33.358281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:33.358292 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:33.358310 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:33.358318 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:33.358373 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.358532 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.358582 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:58:33.359853 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.350639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33665 (local address 127.0.98.2:56157)
0620 08:58:33.351380 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.351433 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.353091 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.353603 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.353612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.353715 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:33.354387 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.354408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.355365 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.355369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.356737 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.356749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.357230 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.357234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.357417 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.357992 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:33.357994 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:33.358045 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:33.358124 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:33.358391 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:33.358400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:33.358405 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.358540 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.358833 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260620 08:58:33.361004 18294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33665
I20260620 08:58:33.361248 18294 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:33.361860 18294 heartbeater.cc:511] Master 127.0.98.62:33665 requested a full tablet report, sending...
I20260620 08:58:33.362995 17974 ts_manager.cc:194] Registered new tserver with Master: 5edd0a478057497b9f35de9c488d2c75 (127.0.98.2:32791)
I20260620 08:58:33.363551 17974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56157
W20260620 08:58:33.463779 18297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:33.464012 18297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:33.464047 18297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:33.468273 18297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:33.468326 18297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:33.468387 18297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:33.473137 18297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:33.474124 18297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:33.475438 18297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:33.484581 18305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:33.484594 18302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:33.484611 18303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:33.484666 18297 server_base.cc:1034] running on GCE node
I20260620 08:58:33.485140 18297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:33.485718 18297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:33.486884 18297 hybrid_clock.cc:648] HybridClock initialized: now 1781945913486856 us; error 39 us; skew 500 ppm
I20260620 08:58:33.488651 18297 webserver.cc:460] Webserver started at http://127.0.98.3:41557/ using document root <none> and password file <none>
I20260620 08:58:33.489158 18297 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:33.489214 18297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:33.489363 18297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:33.491289 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "002b9f483f6743a0811f61a152d07360"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "ecd2cdbab8017f3d4f40e31bd365f216"
server_key_iv: "b63a4f34b7339873aa2ab5fce0aa0bdf"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.491837 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "002b9f483f6743a0811f61a152d07360"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "ecd2cdbab8017f3d4f40e31bd365f216"
server_key_iv: "b63a4f34b7339873aa2ab5fce0aa0bdf"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.495114 18297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 08:58:33.497399 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.498567 18297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:58:33.498667 18297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "002b9f483f6743a0811f61a152d07360"
format_stamp: "Formatted at 2026-06-20 08:58:33 on dist-test-slave-fb1v"
server_key: "ecd2cdbab8017f3d4f40e31bd365f216"
server_key_iv: "b63a4f34b7339873aa2ab5fce0aa0bdf"
server_key_version: "encryptionkey@0"
I20260620 08:58:33.498746 18297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:33.507534 18297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:33.508142 18297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:33.508273 18297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:33.508738 18297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:33.509559 18297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:33.509603 18297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.509639 18297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:33.509657 18297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:33.517915 18297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41971
I20260620 08:58:33.517946 18424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41971 every 8 connection(s)
I20260620 08:58:33.518716 18297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:33.520711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18297
I20260620 08:58:33.520833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:58:33.521116 392 external_mini_cluster.cc:1428] Setting key c6f8e790922b5517656ac931f94fd83c
I20260620 08:58:33.527998 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.520526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45139 (local address 127.0.98.62:33665)
0620 08:58:33.520708 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.520712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.521853 (+ 1141us) server_negotiation.cc:407] Connection header received
0620 08:58:33.523462 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.523465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.523508 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.523579 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:33.524415 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.525101 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.526223 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.526476 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.526556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.527600 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:33.527633 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:33.527636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:33.527650 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:33.527667 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:33.527675 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:33.527728 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.527847 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.527891 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 08:58:33.528791 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.520992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33665 (local address 127.0.98.3:45139)
0620 08:58:33.521722 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.521770 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.523264 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.523753 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.523759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.523840 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:33.524283 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.524295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.525240 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.525243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.526102 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.526111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.526622 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.526625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.526770 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.527345 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:33.527347 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:33.527399 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:33.527473 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:33.527745 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:33.527751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:33.527755 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.527841 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.528047 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260620 08:58:33.529760 18425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33665
I20260620 08:58:33.529964 18425 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:33.530473 18425 heartbeater.cc:511] Master 127.0.98.62:33665 requested a full tablet report, sending...
I20260620 08:58:33.531384 17974 ts_manager.cc:194] Registered new tserver with Master: 002b9f483f6743a0811f61a152d07360 (127.0.98.3:41971)
I20260620 08:58:33.531980 17974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45139
I20260620 08:58:33.534895 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:58:33.540372 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.536771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.0.98.62:33665)
0620 08:58:33.536886 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.536889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.537067 (+ 178us) server_negotiation.cc:407] Connection header received
0620 08:58:33.537138 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.537141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.537222 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.537299 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:33.537569 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.538146 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.539028 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.539231 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.539319 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.540071 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:33.540081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:33.540083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:33.540092 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:33.540110 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:33.540116 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:33.540167 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.540225 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.540267 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:58:33.543031 17974 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:58:33.546038 17974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:58:33.559342 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.553878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46583 (local address 127.0.0.1:60686)
0620 08:58:33.554465 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.554525 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.554642 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.555520 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.555525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.555547 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:33.555698 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.555705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.556745 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.556749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.557942 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.557953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.558827 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.558831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.558942 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.558978 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.559080 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.559195 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":6,"thread_start_us":102,"threads_started":1}
I20260620 08:58:33.559345 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.554488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.0.98.1:46583)
0620 08:58:33.555016 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.555028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.555049 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:58:33.555118 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.555122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.555304 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.555821 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:33.555940 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.556642 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.558077 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.558720 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.558816 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.558997 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.559114 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.559204 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":106,"threads_started":1}
I20260620 08:58:33.559669 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.554073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41971 (local address 127.0.0.1:52786)
0620 08:58:33.554865 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.554876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.554959 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.555563 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.555566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.555588 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:33.555755 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.555760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.557161 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.557165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.558137 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.558146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.559284 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.559287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.559369 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.559390 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.559455 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.559515 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":43,"spinlock_wait_cycles":2304,"thread_start_us":104,"threads_started":1}
I20260620 08:58:33.560094 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.553730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32791 (local address 127.0.0.1:50658)
0620 08:58:33.554465 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.554516 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.554680 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.555581 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.555584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.555597 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:33.555749 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.555756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.557130 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.557133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.558272 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.558281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.559393 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.559396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.559729 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.559835 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.559906 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.559968 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":66,"thread_start_us":78,"threads_started":1}
I20260620 08:58:33.560112 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.554471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.0.98.3:41971)
0620 08:58:33.554964 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.554975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.555011 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:58:33.555080 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.555085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.555282 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.555434 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:33.555868 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.557017 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.558229 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.558960 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.559059 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.559223 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.559872 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.559972 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":117,"threads_started":1}
I20260620 08:58:33.560191 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.553903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.98.2:32791)
0620 08:58:33.554292 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.554304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.554605 (+ 301us) server_negotiation.cc:407] Connection header received
0620 08:58:33.555123 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.555128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.555304 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.555452 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:33.555853 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.557021 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.558382 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.559117 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.559202 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.559344 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.559954 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.560047 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":102,"threads_started":1}
I20260620 08:58:33.562570 18097 tablet_service.cc:1467] Processing CreateTablet for tablet 68d819a293a14b8886b30b766b755794 (DEFAULT_TABLE table=test-table [id=6a975ed8a9bd456998fc25bfb1a4c5d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:33.562569 18359 tablet_service.cc:1467] Processing CreateTablet for tablet 68d819a293a14b8886b30b766b755794 (DEFAULT_TABLE table=test-table [id=6a975ed8a9bd456998fc25bfb1a4c5d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:33.562640 18228 tablet_service.cc:1467] Processing CreateTablet for tablet 68d819a293a14b8886b30b766b755794 (DEFAULT_TABLE table=test-table [id=6a975ed8a9bd456998fc25bfb1a4c5d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:33.563687 18097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d819a293a14b8886b30b766b755794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:33.563687 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d819a293a14b8886b30b766b755794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:33.563689 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d819a293a14b8886b30b766b755794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:33.569505 18442 tablet_bootstrap.cc:492] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Bootstrap starting.
I20260620 08:58:33.570804 18444 tablet_bootstrap.cc:492] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: Bootstrap starting.
I20260620 08:58:33.571163 18443 tablet_bootstrap.cc:492] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Bootstrap starting.
I20260620 08:58:33.572288 18442 tablet_bootstrap.cc:654] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:33.573005 18443 tablet_bootstrap.cc:654] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:33.573155 18442 log.cc:824] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:33.573714 18444 tablet_bootstrap.cc:654] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:33.573695 18443 log.cc:824] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:33.574442 18444 log.cc:824] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:33.575377 18442 tablet_bootstrap.cc:492] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: No bootstrap required, opened a new log
I20260620 08:58:33.575587 18442 ts_tablet_manager.cc:1397] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 08:58:33.575639 18443 tablet_bootstrap.cc:492] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: No bootstrap required, opened a new log
I20260620 08:58:33.575843 18443 ts_tablet_manager.cc:1397] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:58:33.576505 18444 tablet_bootstrap.cc:492] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: No bootstrap required, opened a new log
I20260620 08:58:33.576668 18444 ts_tablet_manager.cc:1397] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 08:58:33.579423 18443 raft_consensus.cc:348] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.579668 18443 raft_consensus.cc:374] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:33.579739 18443 raft_consensus.cc:729] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002b9f483f6743a0811f61a152d07360, State: Initialized, Role: FOLLOWER
I20260620 08:58:33.579911 18442 raft_consensus.cc:348] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.580160 18442 raft_consensus.cc:374] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:33.580250 18442 raft_consensus.cc:729] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edd0a478057497b9f35de9c488d2c75, State: Initialized, Role: FOLLOWER
I20260620 08:58:33.580183 18443 consensus_queue.cc:260] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.580681 18442 consensus_queue.cc:260] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.580665 18444 raft_consensus.cc:348] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.580928 18444 raft_consensus.cc:374] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:33.581022 18444 raft_consensus.cc:729] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08493386270e4d8eac7d9c4b7372e048, State: Initialized, Role: FOLLOWER
I20260620 08:58:33.581100 18443 ts_tablet_manager.cc:1428] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 08:58:33.581355 18294 heartbeater.cc:503] Master 127.0.98.62:33665 was elected leader, sending a full tablet report...
I20260620 08:58:33.581434 18444 consensus_queue.cc:260] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.581602 18442 ts_tablet_manager.cc:1428] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:58:33.582016 18163 heartbeater.cc:503] Master 127.0.98.62:33665 was elected leader, sending a full tablet report...
I20260620 08:58:33.582084 18425 heartbeater.cc:503] Master 127.0.98.62:33665 was elected leader, sending a full tablet report...
I20260620 08:58:33.582305 18444 ts_tablet_manager.cc:1428] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 08:58:33.599258 18295 tablet.cc:2369] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:33.657752 18448 raft_consensus.cc:482] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 08:58:33.657861 18164 tablet.cc:2369] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:33.658025 18448 raft_consensus.cc:504] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.659200 18448 leader_election.cc:288] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5edd0a478057497b9f35de9c488d2c75 (127.0.98.2:32791), 08493386270e4d8eac7d9c4b7372e048 (127.0.98.1:46583)
I20260620 08:58:33.663970 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.659664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60149 (local address 127.0.98.1:46583)
0620 08:58:33.659777 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.659780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.659790 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:58:33.659838 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.659840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.659881 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.659953 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:33.660225 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.661001 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.662653 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.663285 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.663394 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.663452 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.663776 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.663851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:58:33.663964 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.659519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53493 (local address 127.0.98.2:32791)
0620 08:58:33.659673 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.659677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.659945 (+ 268us) server_negotiation.cc:407] Connection header received
0620 08:58:33.660073 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.660075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.660117 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.660171 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:33.660394 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.661093 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.662653 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.663284 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.663402 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.663453 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.663762 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.663839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 08:58:33.663968 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.659419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46583 (local address 127.0.98.3:60149)
0620 08:58:33.659527 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.659542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.659663 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.659989 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.659992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.660005 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:33.660116 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.660121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.661132 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.661135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.662519 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.662525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.663423 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.663427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.663626 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.663661 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.663730 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.663817 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":99}
I20260620 08:58:33.663981 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.659424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32791 (local address 127.0.98.3:53493)
0620 08:58:33.659850 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.659861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:33.659969 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:33.660201 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:33.660203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:33.660219 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:33.660305 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.660309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.661195 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.661198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.662508 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:33.662520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.663423 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.663427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:33.663599 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.663662 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 08:58:33.663735 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:58:33.663820 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":169,"thread_start_us":165,"threads_started":1}
I20260620 08:58:33.664595 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d819a293a14b8886b30b766b755794" candidate_uuid: "002b9f483f6743a0811f61a152d07360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edd0a478057497b9f35de9c488d2c75" is_pre_election: true
I20260620 08:58:33.664776 18117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d819a293a14b8886b30b766b755794" candidate_uuid: "002b9f483f6743a0811f61a152d07360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08493386270e4d8eac7d9c4b7372e048" is_pre_election: true
I20260620 08:58:33.664903 18248 raft_consensus.cc:2454] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002b9f483f6743a0811f61a152d07360 in term 0.
I20260620 08:58:33.665050 18117 raft_consensus.cc:2454] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 002b9f483f6743a0811f61a152d07360 in term 0.
I20260620 08:58:33.665364 18315 leader_election.cc:302] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 002b9f483f6743a0811f61a152d07360, 5edd0a478057497b9f35de9c488d2c75; no voters:
I20260620 08:58:33.665592 18448 raft_consensus.cc:2781] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:58:33.665669 18448 raft_consensus.cc:482] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:58:33.665716 18448 raft_consensus.cc:3037] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:33.666641 18448 raft_consensus.cc:504] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.666963 18448 leader_election.cc:288] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [CANDIDATE]: Term 1 election: Requested vote from peers 5edd0a478057497b9f35de9c488d2c75 (127.0.98.2:32791), 08493386270e4d8eac7d9c4b7372e048 (127.0.98.1:46583)
I20260620 08:58:33.667299 18117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d819a293a14b8886b30b766b755794" candidate_uuid: "002b9f483f6743a0811f61a152d07360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08493386270e4d8eac7d9c4b7372e048"
I20260620 08:58:33.667315 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68d819a293a14b8886b30b766b755794" candidate_uuid: "002b9f483f6743a0811f61a152d07360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edd0a478057497b9f35de9c488d2c75"
I20260620 08:58:33.667428 18248 raft_consensus.cc:3037] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:33.667428 18117 raft_consensus.cc:3037] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:33.668642 18117 raft_consensus.cc:2454] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002b9f483f6743a0811f61a152d07360 in term 1.
I20260620 08:58:33.668642 18248 raft_consensus.cc:2454] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 002b9f483f6743a0811f61a152d07360 in term 1.
I20260620 08:58:33.668941 18314 leader_election.cc:302] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 002b9f483f6743a0811f61a152d07360, 08493386270e4d8eac7d9c4b7372e048; no voters:
I20260620 08:58:33.669128 18448 raft_consensus.cc:2781] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:33.669360 18448 raft_consensus.cc:686] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 1 LEADER]: Becoming Leader. State: Replica: 002b9f483f6743a0811f61a152d07360, State: Running, Role: LEADER
I20260620 08:58:33.669705 18448 consensus_queue.cc:237] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } }
I20260620 08:58:33.672866 17974 catalog_manager.cc:5495] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 reported cstate change: term changed from 0 to 1, leader changed from <none> to 002b9f483f6743a0811f61a152d07360 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "002b9f483f6743a0811f61a152d07360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002b9f483f6743a0811f61a152d07360" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41971 } health_report { overall_health: HEALTHY } } }
I20260620 08:58:33.682751 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.679296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.0.98.3:41971)
0620 08:58:33.679412 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:33.679417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:33.679430 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:58:33.679534 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:33.679537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:33.679589 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:33.679707 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:58:33.679919 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.680421 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.681435 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:33.681657 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:33.681740 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:33.682003 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:58:33.682378 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:58:33.682477 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:33.682556 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:58:33.682607 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:58:33.687999 18117 raft_consensus.cc:1261] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 1 FOLLOWER]: Refusing update from remote peer 002b9f483f6743a0811f61a152d07360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:33.687999 18248 raft_consensus.cc:1261] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 1 FOLLOWER]: Refusing update from remote peer 002b9f483f6743a0811f61a152d07360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:33.688649 18452 consensus_queue.cc:1035] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08493386270e4d8eac7d9c4b7372e048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:33.688858 18448 consensus_queue.cc:1035] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edd0a478057497b9f35de9c488d2c75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:33.695757 18456 mvcc.cc:205] Tried to move back new op lower bound from 7298850462459318272 to 7298850462394023936. Current Snapshot: MvccSnapshot[applied={T|T < 7298850462459318272}]
I20260620 08:58:33.697912 18458 mvcc.cc:205] Tried to move back new op lower bound from 7298850462459318272 to 7298850462394023936. Current Snapshot: MvccSnapshot[applied={T|T < 7298850462459318272}]
I20260620 08:58:33.703393 17974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:33.703688 17974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:33.705924 17974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdd8d798c5f744619bcd77fff48cc3ce: Sending DeleteTablet for 3 replicas of tablet 68d819a293a14b8886b30b766b755794
I20260620 08:58:33.706671 18228 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d819a293a14b8886b30b766b755794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:33 UTC) from {username='slave'} at 127.0.0.1:50658
I20260620 08:58:33.706714 18097 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d819a293a14b8886b30b766b755794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:33 UTC) from {username='slave'} at 127.0.0.1:60686
I20260620 08:58:33.707173 18464 tablet_replica.cc:330] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048: stopping tablet replica
I20260620 08:58:33.707517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18035
I20260620 08:58:33.707530 18464 raft_consensus.cc:2229] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:33.707566 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet 68d819a293a14b8886b30b766b755794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:33 UTC) from {username='slave'} at 127.0.0.1:52786
I20260620 08:58:33.707861 18464 raft_consensus.cc:2258] T 68d819a293a14b8886b30b766b755794 P 08493386270e4d8eac7d9c4b7372e048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:33.709177 18466 tablet_replica.cc:330] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: stopping tablet replica
I20260620 08:58:33.709456 18466 raft_consensus.cc:2229] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:58:33.709834 18466 raft_consensus.cc:2258] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:33.710193 18465 tablet_replica.cc:330] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: stopping tablet replica
I20260620 08:58:33.710481 18465 raft_consensus.cc:2229] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:33.710758 18465 raft_consensus.cc:2258] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:33.712188 18465 ts_tablet_manager.cc:1905] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:33.712245 18466 ts_tablet_manager.cc:1905] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:33.714864 18465 ts_tablet_manager.cc:1918] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 08:58:33.714846 17961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46583 (error 108)
I20260620 08:58:33.714962 18466 ts_tablet_manager.cc:1918] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:33.715032 18465 log.cc:1196] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/68d819a293a14b8886b30b766b755794
I20260620 08:58:33.715152 18466 log.cc:1196] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/68d819a293a14b8886b30b766b755794
I20260620 08:58:33.715368 18465 ts_tablet_manager.cc:1939] T 68d819a293a14b8886b30b766b755794 P 5edd0a478057497b9f35de9c488d2c75: Deleting consensus metadata
I20260620 08:58:33.715510 18466 ts_tablet_manager.cc:1939] T 68d819a293a14b8886b30b766b755794 P 002b9f483f6743a0811f61a152d07360: Deleting consensus metadata
I20260620 08:58:33.715579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18166
I20260620 08:58:33.717141 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.716900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46583 (local address 127.0.0.1:60694)
0620 08:58:33.717015 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.717076 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 08:58:33.717316 17961 catalog_manager.cc:4584] TS 08493386270e4d8eac7d9c4b7372e048 (127.0.98.1:46583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68d819a293a14b8886b30b766b755794: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46583: connect: Connection refused (error 111)
I20260620 08:58:33.717671 17959 catalog_manager.cc:4841] TS 002b9f483f6743a0811f61a152d07360 (127.0.98.3:41971): tablet 68d819a293a14b8886b30b766b755794 (table test-table [id=6a975ed8a9bd456998fc25bfb1a4c5d8]) successfully deleted
I20260620 08:58:33.720912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18297
I20260620 08:58:33.722640 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:33.721116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32791 (local address 127.0.0.1:50660)
0620 08:58:33.722515 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:33.722570 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1318}
I20260620 08:58:33.726032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 17944
2026-06-20T08:58:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:33 dist-test-slave-fb1v krb5kdc[18470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:33 dist-test-slave-fb1v krb5kdc[18470](info): set up 2 sockets
Jun 20 08:58:33 dist-test-slave-fb1v krb5kdc[18470](info): commencing operation
krb5kdc: starting...
W20260620 08:58:34.683264 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:34 dist-test-slave-fb1v krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:58:34.794552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:34.899626 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:34.899892 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:34.899931 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:34.903921 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:34.903972 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:34.903990 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:34.904004 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:34.904021 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:34.909214 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:34.910182 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:34.911127 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:34.919636 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:34.919764 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:34.919994 18483 server_base.cc:1034] running on GCE node
W20260620 08:58:34.920009 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:34.920547 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:34.921586 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:34.922809 18483 hybrid_clock.cc:648] HybridClock initialized: now 1781945914922756 us; error 64 us; skew 500 ppm
Jun 20 08:58:34 dist-test-slave-fb1v krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:34.925701 18483 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:58:34.926527 18483 webserver.cc:460] Webserver started at http://127.0.98.62:37515/ using document root <none> and password file <none>
I20260620 08:58:34.927062 18483 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:34.927119 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:34.927275 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:34.929265 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "b09eacc95db04c17b68b5bd149faa5a6"
format_stamp: "Formatted at 2026-06-20 08:58:34 on dist-test-slave-fb1v"
server_key: "2adc3da30544f8117781caf73e1511fb"
server_key_iv: "a7f682da7ec7a69630c5e7c3298d4b03"
server_key_version: "encryptionkey@0"
I20260620 08:58:34.929751 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "b09eacc95db04c17b68b5bd149faa5a6"
format_stamp: "Formatted at 2026-06-20 08:58:34 on dist-test-slave-fb1v"
server_key: "2adc3da30544f8117781caf73e1511fb"
server_key_iv: "a7f682da7ec7a69630c5e7c3298d4b03"
server_key_version: "encryptionkey@0"
I20260620 08:58:34.933266 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 08:58:34.935474 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:34.936656 18483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 08:58:34.936759 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "b09eacc95db04c17b68b5bd149faa5a6"
format_stamp: "Formatted at 2026-06-20 08:58:34 on dist-test-slave-fb1v"
server_key: "2adc3da30544f8117781caf73e1511fb"
server_key_iv: "a7f682da7ec7a69630c5e7c3298d4b03"
server_key_version: "encryptionkey@0"
I20260620 08:58:34.936843 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:34.968549 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:34.971633 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:34.971788 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:34.979120 18483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37643
I20260620 08:58:34.979144 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37643 every 8 connection(s)
I20260620 08:58:34.980088 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:34.980337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18483
I20260620 08:58:34.980451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:34.980780 392 external_mini_cluster.cc:1428] Setting key 00f617892f6ed23b5dabe0dd143f3bd1
I20260620 08:58:34.983168 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:34.989429 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: Bootstrap starting.
Jun 20 08:58:34 dist-test-slave-fb1v krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:34.992574 18551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:34.993299 18551 log.cc:824] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:34.994751 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:34.982129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.98.62:37643)
0620 08:58:34.982502 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:34.982519 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:34.982560 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:58:34.983177 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:34.983192 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:34.983539 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:34.983847 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:34.984692 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:34.985681 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:34.986814 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:34.987271 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:34.987396 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:34.990144 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:34.990149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:34.990161 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:34.990188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:34.991783 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:34.992486 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:34.992491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:34.992496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:34.992614 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:34.992940 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:34.992943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:34.992944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:34.993337 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:34.993510 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:34.993793 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 08:58:34.994064 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260620 08:58:34.995119 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: No bootstrap required, opened a new log
I20260620 08:58:34.998576 18551 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09eacc95db04c17b68b5bd149faa5a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37643 } }
I20260620 08:58:34.998790 18551 raft_consensus.cc:374] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:34.998853 18551 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09eacc95db04c17b68b5bd149faa5a6, State: Initialized, Role: FOLLOWER
I20260620 08:58:34.999284 18551 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09eacc95db04c17b68b5bd149faa5a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37643 } }
I20260620 08:58:34.999423 18551 raft_consensus.cc:388] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:34.999485 18551 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:34.999584 18551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:35.000468 18551 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09eacc95db04c17b68b5bd149faa5a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37643 } }
I20260620 08:58:35.000803 18551 leader_election.cc:302] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09eacc95db04c17b68b5bd149faa5a6; no voters:
I20260620 08:58:35.001077 18551 leader_election.cc:288] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:35.001197 18556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:35.001544 18556 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [term 1 LEADER]: Becoming Leader. State: Replica: b09eacc95db04c17b68b5bd149faa5a6, State: Running, Role: LEADER
I20260620 08:58:35.001858 18556 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09eacc95db04c17b68b5bd149faa5a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37643 } }
I20260620 08:58:35.002199 18551 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:35.003301 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09eacc95db04c17b68b5bd149faa5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09eacc95db04c17b68b5bd149faa5a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37643 } } }
I20260620 08:58:35.003420 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:35.003753 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09eacc95db04c17b68b5bd149faa5a6. Latest consensus state: current_term: 1 leader_uuid: "b09eacc95db04c17b68b5bd149faa5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09eacc95db04c17b68b5bd149faa5a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37643 } } }
I20260620 08:58:35.003850 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6 [sys.catalog]: This master's current role is: LEADER
W20260620 08:58:35.006456 18571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:58:35.006521 18571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:58:35.006603 18565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:35.007537 18565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:35.014192 18565 catalog_manager.cc:1347] Generated new cluster ID: 73240990d0944aadbb02c44425d05794
I20260620 08:58:35.014307 18565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:35.024291 18565 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:35.025260 18565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:35.033362 18565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: Generated new TSK 0
I20260620 08:58:35.034271 18565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:58:35.070155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37643
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:35.176651 18577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:35.176896 18577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:35.176937 18577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:35.180850 18577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:35.180904 18577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:35.180959 18577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:35.186189 18577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:35.187140 18577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:35.188107 18577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:35.197135 18582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:35.197207 18577 server_base.cc:1034] running on GCE node
W20260620 08:58:35.197175 18583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:35.197141 18585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:35.197773 18577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:35.198365 18577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:35.199525 18577 hybrid_clock.cc:648] HybridClock initialized: now 1781945915199508 us; error 27 us; skew 500 ppm
Jun 20 08:58:35 dist-test-slave-fb1v krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:35.202147 18577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:58:35.202958 18577 webserver.cc:460] Webserver started at http://127.0.98.1:42983/ using document root <none> and password file <none>
I20260620 08:58:35.203510 18577 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:35.203579 18577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:35.203794 18577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:35.205826 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "22270eff66a947c399432965cb1ff015"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "38bab59052a6aa07ae7d08a53558ff27"
server_key_iv: "a578b5ec2430ae222fca6574d33fdabe"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.206333 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22270eff66a947c399432965cb1ff015"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "38bab59052a6aa07ae7d08a53558ff27"
server_key_iv: "a578b5ec2430ae222fca6574d33fdabe"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.209699 18577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:35.211911 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.213052 18577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:58:35.213150 18577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "22270eff66a947c399432965cb1ff015"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "38bab59052a6aa07ae7d08a53558ff27"
server_key_iv: "a578b5ec2430ae222fca6574d33fdabe"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.213234 18577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:35.230315 18577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:35.233321 18577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:35.233492 18577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:35.234047 18577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:35.234834 18577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:35.234881 18577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.234935 18577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:35.234961 18577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.243830 18577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34927
I20260620 08:58:35.243851 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34927 every 8 connection(s)
I20260620 08:58:35.244685 18577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:35.245927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18577
I20260620 08:58:35.246035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:58:35.246292 392 external_mini_cluster.cc:1428] Setting key 12909fba788c802d8457228f1f72d50d
Jun 20 08:58:35 dist-test-slave-fb1v krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:58:35.258548 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.246398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38863 (local address 127.0.98.62:37643)
0620 08:58:35.246536 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.246540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.247386 (+ 846us) server_negotiation.cc:407] Connection header received
0620 08:58:35.248315 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.248318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.248360 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.248434 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:35.249561 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.250254 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.251642 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.251964 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.252078 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.254716 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:35.254721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:35.254724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:35.254752 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:35.256335 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.257085 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.257089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.257090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.257170 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.257571 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.257574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.257575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.257758 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:35.257836 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.258312 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.258414 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:58:35.259510 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.246787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37643 (local address 127.0.98.1:38863)
0620 08:58:35.247241 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.247282 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.248134 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.248621 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.248627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.248966 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:35.249408 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.249420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.250398 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.250402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.251492 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.251502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.252127 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.252130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.252313 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.252954 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:35.252970 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:35.254544 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:35.256458 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:35.256463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:35.256474 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:35.256951 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:35.257290 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:35.257293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:35.257294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:35.257462 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:35.257845 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:35.257850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:35.258109 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.258284 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.258574 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260620 08:58:35.260829 18706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37643
I20260620 08:58:35.261058 18706 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:35.261575 18706 heartbeater.cc:511] Master 127.0.98.62:37643 requested a full tablet report, sending...
I20260620 08:58:35.262985 18515 ts_manager.cc:194] Registered new tserver with Master: 22270eff66a947c399432965cb1ff015 (127.0.98.1:34927)
I20260620 08:58:35.264106 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:58:35.275239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37643
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:35.390060 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:35.390308 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:35.390357 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:35.394526 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:35.394585 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:35.394649 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:35.399873 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:35.400858 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:35.401780 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:35.411638 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:35.411584 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:35.411584 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:35.411741 18711 server_base.cc:1034] running on GCE node
I20260620 08:58:35.412322 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:35.412943 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:35.414113 18711 hybrid_clock.cc:648] HybridClock initialized: now 1781945915414094 us; error 30 us; skew 500 ppm
Jun 20 08:58:35 dist-test-slave-fb1v krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:35.416760 18711 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:58:35.417596 18711 webserver.cc:460] Webserver started at http://127.0.98.2:45125/ using document root <none> and password file <none>
I20260620 08:58:35.418125 18711 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:35.418188 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:35.418366 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:35.420359 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e5d0b53c94a43deb72384073cd1d605"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "521db486401046826c6d17a0b0b5a11d"
server_key_iv: "238dec157621c31a9285ccdec7752a83"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.420868 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e5d0b53c94a43deb72384073cd1d605"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "521db486401046826c6d17a0b0b5a11d"
server_key_iv: "238dec157621c31a9285ccdec7752a83"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.424320 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:35.426698 18726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.427935 18711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:35.428051 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "1e5d0b53c94a43deb72384073cd1d605"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "521db486401046826c6d17a0b0b5a11d"
server_key_iv: "238dec157621c31a9285ccdec7752a83"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.428145 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:35.443332 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:35.446450 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:35.446648 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:35.447253 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:35.448130 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:35.448182 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.448239 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:35.448271 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.457270 18711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43783
I20260620 08:58:35.457290 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43783 every 8 connection(s)
I20260620 08:58:35.458137 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:35.461535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18711
I20260620 08:58:35.461663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:58:35.461946 392 external_mini_cluster.cc:1428] Setting key 78379eac6a3a6ca846473d8a9a9f8b37
Jun 20 08:58:35 dist-test-slave-fb1v krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:58:35.472222 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.459944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35727 (local address 127.0.98.62:37643)
0620 08:58:35.460072 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.460075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.460860 (+ 785us) server_negotiation.cc:407] Connection header received
0620 08:58:35.461918 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.461922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.462004 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.462184 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:35.463353 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.464055 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.465779 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.466179 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.466299 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.468827 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:35.468831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:35.468835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:35.468858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:35.470132 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.470721 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.470724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.470725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.470806 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.471155 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.471157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.471158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.471359 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:35.471444 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.471991 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.472101 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:58:35.473001 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.460286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37643 (local address 127.0.98.2:35727)
0620 08:58:35.460728 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.460768 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.461620 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.462341 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.462348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.462757 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:35.463208 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.463220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.464301 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.464305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.465618 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.465628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.466352 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.466355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.466510 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.467078 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:35.467094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:35.468638 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:35.470247 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:35.470253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:35.470267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:35.470605 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:35.470901 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:35.470904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:35.470906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:35.471077 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:35.471463 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:35.471469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:35.471765 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.471924 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.472174 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:58:35.474218 18840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37643
I20260620 08:58:35.474471 18840 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:35.475064 18840 heartbeater.cc:511] Master 127.0.98.62:37643 requested a full tablet report, sending...
I20260620 08:58:35.476177 18515 ts_manager.cc:194] Registered new tserver with Master: 1e5d0b53c94a43deb72384073cd1d605 (127.0.98.2:43783)
I20260620 08:58:35.476784 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:58:35.490417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37643
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:35.598582 18845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:35.598824 18845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:35.598865 18845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:35.602800 18845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:35.602857 18845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:35.602922 18845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:35.608218 18845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:35.609185 18845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:35.610112 18845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:35.619141 18851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:35.619136 18853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:35.619144 18850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:35.619434 18845 server_base.cc:1034] running on GCE node
I20260620 08:58:35.619817 18845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:35.620368 18845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:35.621515 18845 hybrid_clock.cc:648] HybridClock initialized: now 1781945915621501 us; error 26 us; skew 500 ppm
Jun 20 08:58:35 dist-test-slave-fb1v krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:35.624056 18845 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:58:35.624876 18845 webserver.cc:460] Webserver started at http://127.0.98.3:36683/ using document root <none> and password file <none>
I20260620 08:58:35.625370 18845 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:35.625427 18845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:35.625607 18845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:35.627538 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "67b7dcbe2f084162a14d639dd6d1cec0"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "5380153bccbfbad119509c90c4a495f9"
server_key_iv: "d29b711ae6ac281aa086125fbfc52684"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.628086 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67b7dcbe2f084162a14d639dd6d1cec0"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "5380153bccbfbad119509c90c4a495f9"
server_key_iv: "d29b711ae6ac281aa086125fbfc52684"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.631500 18845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:35.633716 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.634820 18845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:35.634922 18845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "67b7dcbe2f084162a14d639dd6d1cec0"
format_stamp: "Formatted at 2026-06-20 08:58:35 on dist-test-slave-fb1v"
server_key: "5380153bccbfbad119509c90c4a495f9"
server_key_iv: "d29b711ae6ac281aa086125fbfc52684"
server_key_version: "encryptionkey@0"
I20260620 08:58:35.635010 18845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:35.648052 18845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:35.650795 18845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:35.650962 18845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:35.651504 18845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:35.652307 18845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:35.652352 18845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.652403 18845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:35.652432 18845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:35.660753 18845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42137
I20260620 08:58:35.660786 18973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42137 every 8 connection(s)
I20260620 08:58:35.661616 18845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:35.667351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 18845
I20260620 08:58:35.667457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:58:35.667743 392 external_mini_cluster.cc:1428] Setting key 79aa3f11e69590fb337ab6baee8ebfd3
Jun 20 08:58:35 dist-test-slave-fb1v krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:35.674743 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.663284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49595 (local address 127.0.98.62:37643)
0620 08:58:35.663413 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.663417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.664456 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 08:58:35.665326 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.665329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.665388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.665458 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:35.666557 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.667195 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.668520 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.668796 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.668902 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.671398 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:35.671401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:35.671403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:35.671425 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:35.672723 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.673128 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.673131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.673132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.673199 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.673475 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.673478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.673479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.673856 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:35.674020 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.674507 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.674627 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:58:35.675498 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.663634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37643 (local address 127.0.98.3:49595)
0620 08:58:35.664298 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.664337 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.665129 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.665638 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.665645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.665971 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:35.666416 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.666428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.667336 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.667339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.668408 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.668417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.668975 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.668980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.669153 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.669753 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:35.669773 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:35.671230 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:35.672805 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:35.672809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:35.672819 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:35.673056 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:35.673259 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:35.673261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:35.673262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:35.673376 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:35.674022 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:35.674040 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:35.674259 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.674416 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.674659 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":104,"threads_started":1}
I20260620 08:58:35.676674 18974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37643
I20260620 08:58:35.676903 18974 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:35.677356 18974 heartbeater.cc:511] Master 127.0.98.62:37643 requested a full tablet report, sending...
I20260620 08:58:35.678397 18515 ts_manager.cc:194] Registered new tserver with Master: 67b7dcbe2f084162a14d639dd6d1cec0 (127.0.98.3:42137)
I20260620 08:58:35.678933 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49595
I20260620 08:58:35.681350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:58:35.690433 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.683150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.0.98.62:37643)
0620 08:58:35.683292 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.683296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.683436 (+ 140us) server_negotiation.cc:407] Connection header received
0620 08:58:35.683556 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.683559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.683598 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.683691 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:35.684330 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.684859 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.685661 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.685868 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.685963 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.687428 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:35.687431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:35.687433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:35.687455 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:35.688645 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.689116 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.689119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.689121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.689189 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:35.689603 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:35.689606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:35.689607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:35.689801 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:35.689943 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.690207 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.690294 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:58:35.693547 18515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:58:35.696779 18515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:58:35.711014 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.705179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.0.98.1:34927)
0620 08:58:35.706146 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.706157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.706179 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:58:35.706251 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.706256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.706457 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.706606 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:35.706908 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.708000 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.709582 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.710211 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.710324 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.710488 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.710720 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.710834 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":99,"threads_started":1}
I20260620 08:58:35.711150 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.705192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42137 (local address 127.0.0.1:46152)
0620 08:58:35.706153 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.706168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.706247 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.707055 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.707058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.707072 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:35.707154 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.707158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.708454 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.708457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.709644 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.709655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.710680 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.710684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.710781 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.710817 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.710899 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.711016 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":91,"threads_started":1}
I20260620 08:58:35.711155 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.705905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.0.98.3:42137)
0620 08:58:35.706517 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.706532 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.706554 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:58:35.706621 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.706625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.706786 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.706915 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:35.707268 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.708329 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.709776 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.710567 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.710665 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.710789 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.710924 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.711007 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":125,"threads_started":1}
I20260620 08:58:35.711772 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.705028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34927 (local address 127.0.0.1:56448)
0620 08:58:35.705680 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.705728 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.705853 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.706605 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.706611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.706644 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:35.706771 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.706777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.708127 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.708130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.709461 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.709470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.710339 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.710343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.710498 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.710542 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.711536 (+ 994us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.711637 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":24,"thread_start_us":88,"threads_started":1}
I20260620 08:58:35.711964 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.705709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43783 (local address 127.0.0.1:39198)
0620 08:58:35.707332 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.707341 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.707410 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.707810 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.707813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.707829 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:35.707923 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.707929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.709312 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.709315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.710375 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.710386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.711556 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.711559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.711677 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.711700 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.711771 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.711835 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1576,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260620 08:58:35.712359 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.705909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.98.2:43783)
0620 08:58:35.706896 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.706906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.707421 (+ 515us) server_negotiation.cc:407] Connection header received
0620 08:58:35.707510 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.707515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.707701 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.707820 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:35.708015 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.709171 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.710511 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.711276 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.711371 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.711494 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.711851 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.712196 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":98,"threads_started":1}
I20260620 08:58:35.714459 18774 tablet_service.cc:1467] Processing CreateTablet for tablet c8123df0ad9c4033ae7de9a8c09a0100 (DEFAULT_TABLE table=test-table [id=f88dbb8d62ef4894bb7b1d93fe3a64bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:35.714318 18908 tablet_service.cc:1467] Processing CreateTablet for tablet c8123df0ad9c4033ae7de9a8c09a0100 (DEFAULT_TABLE table=test-table [id=f88dbb8d62ef4894bb7b1d93fe3a64bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:35.714628 18640 tablet_service.cc:1467] Processing CreateTablet for tablet c8123df0ad9c4033ae7de9a8c09a0100 (DEFAULT_TABLE table=test-table [id=f88dbb8d62ef4894bb7b1d93fe3a64bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:35.715385 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8123df0ad9c4033ae7de9a8c09a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:35.715451 18908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8123df0ad9c4033ae7de9a8c09a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:35.715474 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8123df0ad9c4033ae7de9a8c09a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:35.721369 18991 tablet_bootstrap.cc:492] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Bootstrap starting.
I20260620 08:58:35.722796 18992 tablet_bootstrap.cc:492] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: Bootstrap starting.
I20260620 08:58:35.723917 18993 tablet_bootstrap.cc:492] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Bootstrap starting.
I20260620 08:58:35.723963 18991 tablet_bootstrap.cc:654] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:35.724671 18991 log.cc:824] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:35.724695 18992 tablet_bootstrap.cc:654] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:35.725790 18992 log.cc:824] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:35.726184 18993 tablet_bootstrap.cc:654] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:35.726449 18991 tablet_bootstrap.cc:492] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: No bootstrap required, opened a new log
I20260620 08:58:35.726636 18991 ts_tablet_manager.cc:1397] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:58:35.727475 18993 log.cc:824] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:35.727715 18992 tablet_bootstrap.cc:492] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: No bootstrap required, opened a new log
I20260620 08:58:35.727941 18992 ts_tablet_manager.cc:1397] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:58:35.729386 18993 tablet_bootstrap.cc:492] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: No bootstrap required, opened a new log
I20260620 08:58:35.729568 18993 ts_tablet_manager.cc:1397] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 08:58:35.729925 18991 raft_consensus.cc:348] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.730156 18991 raft_consensus.cc:374] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:35.730238 18991 raft_consensus.cc:729] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5d0b53c94a43deb72384073cd1d605, State: Initialized, Role: FOLLOWER
I20260620 08:58:35.730661 18991 consensus_queue.cc:260] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.731304 18840 heartbeater.cc:503] Master 127.0.98.62:37643 was elected leader, sending a full tablet report...
I20260620 08:58:35.731621 18991 ts_tablet_manager.cc:1428] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 08:58:35.732192 18992 raft_consensus.cc:348] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.732427 18992 raft_consensus.cc:374] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:35.732525 18992 raft_consensus.cc:729] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22270eff66a947c399432965cb1ff015, State: Initialized, Role: FOLLOWER
I20260620 08:58:35.732963 18992 consensus_queue.cc:260] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.733121 18993 raft_consensus.cc:348] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.733453 18993 raft_consensus.cc:374] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:35.733542 18993 raft_consensus.cc:729] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b7dcbe2f084162a14d639dd6d1cec0, State: Initialized, Role: FOLLOWER
I20260620 08:58:35.733618 18706 heartbeater.cc:503] Master 127.0.98.62:37643 was elected leader, sending a full tablet report...
I20260620 08:58:35.733965 18992 ts_tablet_manager.cc:1428] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:58:35.734038 18993 consensus_queue.cc:260] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.734997 18974 heartbeater.cc:503] Master 127.0.98.62:37643 was elected leader, sending a full tablet report...
I20260620 08:58:35.735708 18993 ts_tablet_manager.cc:1428] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 08:58:35.745903 18707 tablet.cc:2369] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:35.823899 18997 raft_consensus.cc:482] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:35.824149 18997 raft_consensus.cc:504] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.825415 18997 leader_election.cc:288] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22270eff66a947c399432965cb1ff015 (127.0.98.1:34927), 67b7dcbe2f084162a14d639dd6d1cec0 (127.0.98.3:42137)
I20260620 08:58:35.830225 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.825746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51153 (local address 127.0.98.1:34927)
0620 08:58:35.825882 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.825887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.825904 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:58:35.826007 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.826010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.826061 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.826142 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:35.826455 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.827217 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.828905 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.829531 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.829648 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.829712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.830016 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.830096 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:58:35.830225 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.825803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58925 (local address 127.0.98.3:42137)
0620 08:58:35.825914 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.825918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.826198 (+ 280us) server_negotiation.cc:407] Connection header received
0620 08:58:35.826310 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.826312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.826357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.826410 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:35.826633 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.827318 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.828905 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.829545 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.829655 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.829711 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.830016 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.830103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:58:35.830255 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.825679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34927 (local address 127.0.98.2:51153)
0620 08:58:35.825790 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.825804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.825909 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.826179 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.826182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.826227 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:35.826348 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.826353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.827354 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.827357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.828765 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.828775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.829677 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.829681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.829851 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.829894 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.829971 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.830084 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":151,"spinlock_wait_cycles":46336}
I20260620 08:58:35.830262 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.825689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42137 (local address 127.0.98.2:58925)
0620 08:58:35.826115 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.826126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:35.826216 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:35.826439 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:35.826442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:35.826460 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:35.826548 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.826552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.827430 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.827433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.828765 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:35.828775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.829680 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.829683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:35.829854 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.829891 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:58:35.829975 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 08:58:35.830091 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":277,"thread_start_us":117,"threads_started":1}
I20260620 08:58:35.831020 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8123df0ad9c4033ae7de9a8c09a0100" candidate_uuid: "1e5d0b53c94a43deb72384073cd1d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22270eff66a947c399432965cb1ff015" is_pre_election: true
I20260620 08:58:35.831111 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8123df0ad9c4033ae7de9a8c09a0100" candidate_uuid: "1e5d0b53c94a43deb72384073cd1d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" is_pre_election: true
I20260620 08:58:35.831344 18928 raft_consensus.cc:2454] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5d0b53c94a43deb72384073cd1d605 in term 0.
I20260620 08:58:35.831344 18660 raft_consensus.cc:2454] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5d0b53c94a43deb72384073cd1d605 in term 0.
I20260620 08:58:35.831844 18728 leader_election.cc:302] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e5d0b53c94a43deb72384073cd1d605, 22270eff66a947c399432965cb1ff015, 67b7dcbe2f084162a14d639dd6d1cec0; no voters:
I20260620 08:58:35.832093 18997 raft_consensus.cc:2781] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:58:35.832183 18997 raft_consensus.cc:482] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:58:35.832233 18997 raft_consensus.cc:3037] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:35.833381 18997 raft_consensus.cc:504] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.833701 18997 leader_election.cc:288] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [CANDIDATE]: Term 1 election: Requested vote from peers 22270eff66a947c399432965cb1ff015 (127.0.98.1:34927), 67b7dcbe2f084162a14d639dd6d1cec0 (127.0.98.3:42137)
I20260620 08:58:35.834018 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8123df0ad9c4033ae7de9a8c09a0100" candidate_uuid: "1e5d0b53c94a43deb72384073cd1d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22270eff66a947c399432965cb1ff015"
I20260620 08:58:35.834041 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8123df0ad9c4033ae7de9a8c09a0100" candidate_uuid: "1e5d0b53c94a43deb72384073cd1d605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b7dcbe2f084162a14d639dd6d1cec0"
I20260620 08:58:35.834148 18660 raft_consensus.cc:3037] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:35.834144 18928 raft_consensus.cc:3037] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:35.835317 18928 raft_consensus.cc:2454] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5d0b53c94a43deb72384073cd1d605 in term 1.
I20260620 08:58:35.835317 18660 raft_consensus.cc:2454] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5d0b53c94a43deb72384073cd1d605 in term 1.
I20260620 08:58:35.835598 18728 leader_election.cc:302] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5d0b53c94a43deb72384073cd1d605, 67b7dcbe2f084162a14d639dd6d1cec0; no voters:
I20260620 08:58:35.835776 18997 raft_consensus.cc:2781] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:35.836030 18997 raft_consensus.cc:686] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5d0b53c94a43deb72384073cd1d605, State: Running, Role: LEADER
I20260620 08:58:35.836493 18997 consensus_queue.cc:237] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } }
I20260620 08:58:35.839330 18515 catalog_manager.cc:5495] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5d0b53c94a43deb72384073cd1d605 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e5d0b53c94a43deb72384073cd1d605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5d0b53c94a43deb72384073cd1d605" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43783 } health_report { overall_health: HEALTHY } } }
I20260620 08:58:35.872663 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.868422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.0.98.2:43783)
0620 08:58:35.868625 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:35.868629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:35.868646 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:58:35.868893 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:35.868904 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:35.868965 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:35.869063 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:58:35.869305 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.869826 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.870980 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:35.871209 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:35.871300 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:35.871791 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:58:35.872223 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:58:35.872343 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:35.872512 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 08:58:35.872547 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 08:58:35.878717 18660 raft_consensus.cc:1261] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 1 FOLLOWER]: Refusing update from remote peer 1e5d0b53c94a43deb72384073cd1d605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:35.878718 18928 raft_consensus.cc:1261] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 1 FOLLOWER]: Refusing update from remote peer 1e5d0b53c94a43deb72384073cd1d605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:35.879340 19001 consensus_queue.cc:1035] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b7dcbe2f084162a14d639dd6d1cec0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:35.879528 18997 consensus_queue.cc:1035] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22270eff66a947c399432965cb1ff015" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:35.894953 18515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:35.895129 18515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:35.897379 18515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b09eacc95db04c17b68b5bd149faa5a6: Sending DeleteTablet for 3 replicas of tablet c8123df0ad9c4033ae7de9a8c09a0100
I20260620 08:58:35.898325 18640 tablet_service.cc:1514] Processing DeleteTablet for tablet c8123df0ad9c4033ae7de9a8c09a0100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56448
I20260620 08:58:35.898388 18908 tablet_service.cc:1514] Processing DeleteTablet for tablet c8123df0ad9c4033ae7de9a8c09a0100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46152
I20260620 08:58:35.898880 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet c8123df0ad9c4033ae7de9a8c09a0100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39198
I20260620 08:58:35.899180 19015 tablet_replica.cc:330] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: stopping tablet replica
I20260620 08:58:35.899672 19014 tablet_replica.cc:330] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015: stopping tablet replica
I20260620 08:58:35.899729 19015 raft_consensus.cc:2229] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:35.900022 19014 raft_consensus.cc:2229] T c8123df0ad9c4033ae7de9a8c09a0100 P 22270eff66a947c399432965cb1ff015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:35.900069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18577
I20260620 08:58:35.900722 19015 raft_consensus.cc:2258] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:35.900842 19016 tablet_replica.cc:330] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: stopping tablet replica
I20260620 08:58:35.902400 19016 raft_consensus.cc:2229] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:58:35.902889 19016 raft_consensus.cc:2258] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:35.903234 19015 ts_tablet_manager.cc:1905] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:35.904733 19016 ts_tablet_manager.cc:1905] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:35.905983 19015 ts_tablet_manager.cc:1918] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:35.906078 19015 log.cc:1196] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/c8123df0ad9c4033ae7de9a8c09a0100
I20260620 08:58:35.906358 19015 ts_tablet_manager.cc:1939] T c8123df0ad9c4033ae7de9a8c09a0100 P 67b7dcbe2f084162a14d639dd6d1cec0: Deleting consensus metadata
I20260620 08:58:35.907589 18500 catalog_manager.cc:4841] TS 67b7dcbe2f084162a14d639dd6d1cec0 (127.0.98.3:42137): tablet c8123df0ad9c4033ae7de9a8c09a0100 (table test-table [id=f88dbb8d62ef4894bb7b1d93fe3a64bd]) successfully deleted
I20260620 08:58:35.908331 19016 ts_tablet_manager.cc:1918] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:35.908491 19016 log.cc:1196] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/c8123df0ad9c4033ae7de9a8c09a0100
W20260620 08:58:35.908607 18501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34927 (error 108)
I20260620 08:58:35.908867 19016 ts_tablet_manager.cc:1939] T c8123df0ad9c4033ae7de9a8c09a0100 P 1e5d0b53c94a43deb72384073cd1d605: Deleting consensus metadata
I20260620 08:58:35.909578 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.909314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34927 (local address 127.0.0.1:56462)
0620 08:58:35.909431 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.909495 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:58:35.909602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18711
W20260620 08:58:35.909814 18501 catalog_manager.cc:4584] TS 22270eff66a947c399432965cb1ff015 (127.0.98.1:34927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8123df0ad9c4033ae7de9a8c09a0100: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34927: connect: Connection refused (error 111)
I20260620 08:58:35.914753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18845
I20260620 08:58:35.916165 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:35.914991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43783 (local address 127.0.0.1:39220)
0620 08:58:35.916050 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:35.916098 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":983}
I20260620 08:58:35.919497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 18483
2026-06-20T08:58:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2193 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T08:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:35Z Disabled control of system clock
I20260620 08:58:35.947000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36735
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:58:36.054821 19021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:36.055075 19021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:36.055109 19021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:36.059067 19021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:36.059127 19021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:36.059141 19021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:36.059156 19021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:36.059170 19021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:36.063920 19021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:36.064957 19021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:36.066263 19021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:36.074445 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:36.074443 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:36.074512 19021 server_base.cc:1034] running on GCE node
W20260620 08:58:36.074446 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:36.075201 19021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:36.076860 19021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:36.078040 19021 hybrid_clock.cc:648] HybridClock initialized: now 1781945916078010 us; error 40 us; skew 500 ppm
I20260620 08:58:36.080214 19021 webserver.cc:460] Webserver started at http://127.0.98.62:37171/ using document root <none> and password file <none>
I20260620 08:58:36.080852 19021 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:36.080947 19021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:36.081212 19021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:36.084089 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "bf12a2288f0d48bc85e2e25093c0bb05"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "93b0c2f81857d5473bfa68f954a53f61"
server_key_iv: "545c77c104decf34c6b1a423979b5299"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.084722 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf12a2288f0d48bc85e2e25093c0bb05"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "93b0c2f81857d5473bfa68f954a53f61"
server_key_iv: "545c77c104decf34c6b1a423979b5299"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.088587 19021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 08:58:36.091325 19035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.092583 19021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 08:58:36.092702 19021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "bf12a2288f0d48bc85e2e25093c0bb05"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "93b0c2f81857d5473bfa68f954a53f61"
server_key_iv: "545c77c104decf34c6b1a423979b5299"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.092797 19021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:36.105193 19021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:36.105813 19021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:36.105981 19021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:36.112294 19021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36735
I20260620 08:58:36.112314 19087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36735 every 8 connection(s)
I20260620 08:58:36.113157 19021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:36.116087 19088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:36.121194 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05: Bootstrap starting.
I20260620 08:58:36.122294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19021
I20260620 08:58:36.122402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:36.122650 392 external_mini_cluster.cc:1428] Setting key b99ae8d2327dff6d11d042d37e8f154b
I20260620 08:58:36.123641 19088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:36.124397 19088 log.cc:824] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:36.126133 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05: No bootstrap required, opened a new log
I20260620 08:58:36.129501 19088 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260620 08:58:36.129707 19088 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:36.129740 19088 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf12a2288f0d48bc85e2e25093c0bb05, State: Initialized, Role: FOLLOWER
I20260620 08:58:36.130156 19088 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260620 08:58:36.130304 19088 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:36.130367 19088 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:36.130455 19088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:36.131054 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.123673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.0.98.62:36735)
0620 08:58:36.124291 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.124305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.124337 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:58:36.124967 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.124992 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.125300 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.125587 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:36.126013 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.126963 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.127880 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.128365 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.128510 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.129321 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:36.129326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:36.129338 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:36.129698 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:36.129766 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:36.129792 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:36.129877 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.130027 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.130271 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260620 08:58:36.131347 19088 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260620 08:58:36.131680 19088 leader_election.cc:302] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf12a2288f0d48bc85e2e25093c0bb05; no voters:
I20260620 08:58:36.131948 19088 leader_election.cc:288] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:36.132074 19093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:36.132246 19093 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [term 1 LEADER]: Becoming Leader. State: Replica: bf12a2288f0d48bc85e2e25093c0bb05, State: Running, Role: LEADER
I20260620 08:58:36.132519 19093 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260620 08:58:36.133132 19088 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:36.133632 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf12a2288f0d48bc85e2e25093c0bb05. Latest consensus state: current_term: 1 leader_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } } }
I20260620 08:58:36.133752 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:36.134011 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf12a2288f0d48bc85e2e25093c0bb05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } } }
I20260620 08:58:36.134124 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:36.134127 19098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:36.136824 19098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:36.142112 19098 catalog_manager.cc:1347] Generated new cluster ID: 7e25a4a5a41044be96f07c6fc56c77e0
I20260620 08:58:36.142180 19098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:36.150439 19098 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:36.151577 19098 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:36.163785 19098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05: Generated new TSK 0
I20260620 08:58:36.164637 19098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:36.171890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--builtin_ntp_servers=127.0.98.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:58:36.277899 19112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:36.278129 19112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:36.278167 19112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:36.282195 19112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:36.282249 19112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:36.282303 19112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:36.286991 19112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:36.288066 19112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:36.289342 19112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:58:36.299203 19112 server_base.cc:1034] running on GCE node
W20260620 08:58:36.299180 19117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:36.299160 19120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:36.299379 19118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:36.299784 19112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:36.300405 19112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:36.301582 19112 hybrid_clock.cc:648] HybridClock initialized: now 1781945916301569 us; error 26 us; skew 500 ppm
I20260620 08:58:36.303318 19112 webserver.cc:460] Webserver started at http://127.0.98.1:46475/ using document root <none> and password file <none>
I20260620 08:58:36.303903 19112 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:36.303970 19112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:36.304157 19112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:36.306098 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "8859f243a86c49ffa7a7be73d01cabcb"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "8844ac2bd9602d9768170994dd89b4d8"
server_key_iv: "86283419a5422b63c9f6f85e8c374769"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.306598 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8859f243a86c49ffa7a7be73d01cabcb"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "8844ac2bd9602d9768170994dd89b4d8"
server_key_iv: "86283419a5422b63c9f6f85e8c374769"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.310032 19112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:36.312736 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.313984 19112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:36.314112 19112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "8859f243a86c49ffa7a7be73d01cabcb"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "8844ac2bd9602d9768170994dd89b4d8"
server_key_iv: "86283419a5422b63c9f6f85e8c374769"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.314208 19112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:36.326066 19112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:36.326659 19112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:36.326820 19112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:36.327368 19112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:36.328261 19112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:36.328307 19112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.328357 19112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:36.328387 19112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.336510 19112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45695
I20260620 08:58:36.336536 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45695 every 8 connection(s)
I20260620 08:58:36.337345 19112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:36.337924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19112
I20260620 08:58:36.338073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:58:36.338374 392 external_mini_cluster.cc:1428] Setting key a26e8601f34a07bd423d23bef7a39ef2
I20260620 08:58:36.340492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--builtin_ntp_servers=127.0.98.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:58:36.347920 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.339197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48025 (local address 127.0.98.62:36735)
0620 08:58:36.339311 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.339314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.340633 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 08:58:36.342351 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.342355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.342430 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.342523 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:36.343592 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.344229 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.345892 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.346115 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.346196 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.347452 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:36.347456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:36.347459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:36.347473 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:36.347493 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:36.347503 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:36.347569 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.347754 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.347809 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:58:36.349036 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.339719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36735 (local address 127.0.98.1:48025)
0620 08:58:36.340490 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.340542 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.342136 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.342714 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.342721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.342808 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:36.343418 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.343435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.344367 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.344370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.345759 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.345771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.346259 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.346264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.346471 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.347133 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:36.347135 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:36.347189 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:36.347295 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:36.347584 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:36.347593 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:36.347599 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.347760 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.348048 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260620 08:58:36.350186 19240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36735
I20260620 08:58:36.350425 19240 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:36.351028 19240 heartbeater.cc:511] Master 127.0.98.62:36735 requested a full tablet report, sending...
I20260620 08:58:36.352509 19052 ts_manager.cc:194] Registered new tserver with Master: 8859f243a86c49ffa7a7be73d01cabcb (127.0.98.1:45695)
I20260620 08:58:36.353621 19052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48025
W20260620 08:58:36.448367 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:36.448622 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:36.448674 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:36.452608 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:36.452661 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:36.452733 19243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:36.457351 19243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:36.458341 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:36.459621 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:36.468582 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:36.468576 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:36.468582 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:36.468873 19243 server_base.cc:1034] running on GCE node
I20260620 08:58:36.469305 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:36.469867 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:36.471038 19243 hybrid_clock.cc:648] HybridClock initialized: now 1781945916470999 us; error 53 us; skew 500 ppm
I20260620 08:58:36.472812 19243 webserver.cc:460] Webserver started at http://127.0.98.2:43961/ using document root <none> and password file <none>
I20260620 08:58:36.473356 19243 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:36.473418 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:36.473594 19243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:36.475488 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "d44988682bda4e53947eae7f578b6b7e"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "cb91df35b5cc9526184438757c3e3ca2"
server_key_iv: "51b93b392a059fc21d4aab3dbaa24676"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.476033 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d44988682bda4e53947eae7f578b6b7e"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "cb91df35b5cc9526184438757c3e3ca2"
server_key_iv: "51b93b392a059fc21d4aab3dbaa24676"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.479456 19243 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:36.481788 19257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.483117 19243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:36.483235 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "d44988682bda4e53947eae7f578b6b7e"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "cb91df35b5cc9526184438757c3e3ca2"
server_key_iv: "51b93b392a059fc21d4aab3dbaa24676"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.483333 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:36.502897 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:36.503923 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:36.504124 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:36.504737 19243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:36.505687 19243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:36.505734 19243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.505786 19243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:36.505815 19243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.514068 19243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35209
I20260620 08:58:36.514210 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35209 every 8 connection(s)
I20260620 08:58:36.514948 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:36.517092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19243
I20260620 08:58:36.517196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:58:36.517455 392 external_mini_cluster.cc:1428] Setting key e1bbf51f9fe6bf0c326e125f56141688
I20260620 08:58:36.519239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--builtin_ntp_servers=127.0.98.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:58:36.523898 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.516718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40027 (local address 127.0.98.62:36735)
0620 08:58:36.516858 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.516863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.517937 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 08:58:36.519343 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.519346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.519398 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.519475 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:36.520337 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.520899 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.522100 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.522327 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.522456 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.523433 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:36.523437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:36.523440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:36.523453 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:36.523473 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:36.523481 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:36.523546 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.523692 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.523756 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:58:36.524698 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.517079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36735 (local address 127.0.98.2:40027)
0620 08:58:36.517781 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.517830 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.519148 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.519627 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.519636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.519722 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:36.520197 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.520210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.521013 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.521022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.521977 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.521987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.522490 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.522495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.522660 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.523200 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:36.523202 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:36.523240 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:36.523298 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:36.523547 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:36.523555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:36.523560 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.523691 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.523949 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260620 08:58:36.525676 19371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36735
I20260620 08:58:36.525885 19371 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:36.526345 19371 heartbeater.cc:511] Master 127.0.98.62:36735 requested a full tablet report, sending...
I20260620 08:58:36.527302 19052 ts_manager.cc:194] Registered new tserver with Master: d44988682bda4e53947eae7f578b6b7e (127.0.98.2:35209)
I20260620 08:58:36.528076 19052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40027
W20260620 08:58:36.631793 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:36.632074 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:36.632119 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:36.636483 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:36.636552 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:36.636610 19374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:36.641785 19374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:36.643388 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:36.645483 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:36.654886 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:36.655017 19374 server_base.cc:1034] running on GCE node
W20260620 08:58:36.654886 19379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:36.654948 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:36.655555 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:36.656190 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:36.657357 19374 hybrid_clock.cc:648] HybridClock initialized: now 1781945916657341 us; error 32 us; skew 500 ppm
I20260620 08:58:36.659901 19374 webserver.cc:460] Webserver started at http://127.0.98.3:46413/ using document root <none> and password file <none>
I20260620 08:58:36.660929 19374 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:36.661026 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:36.661303 19374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:36.664485 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "448137f1ea7a48a2ab00e73fb0cf691d"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "eacdd02d7a718aad308f42b5cd385b91"
server_key_iv: "c7ffcf728e9105ce653e28cbcc72430f"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.665169 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "448137f1ea7a48a2ab00e73fb0cf691d"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "eacdd02d7a718aad308f42b5cd385b91"
server_key_iv: "c7ffcf728e9105ce653e28cbcc72430f"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.668932 19374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 08:58:36.671422 19388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.672686 19374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:36.672801 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "448137f1ea7a48a2ab00e73fb0cf691d"
format_stamp: "Formatted at 2026-06-20 08:58:36 on dist-test-slave-fb1v"
server_key: "eacdd02d7a718aad308f42b5cd385b91"
server_key_iv: "c7ffcf728e9105ce653e28cbcc72430f"
server_key_version: "encryptionkey@0"
I20260620 08:58:36.672904 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:36.689082 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:36.689761 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:36.689924 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:36.690487 19374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:36.691387 19374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:36.691432 19374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.691483 19374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:36.691512 19374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:36.699687 19374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42857
I20260620 08:58:36.699740 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42857 every 8 connection(s)
I20260620 08:58:36.700554 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:36.705837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19374
I20260620 08:58:36.705986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:58:36.706336 392 external_mini_cluster.cc:1428] Setting key c0e7fa07505ba0871aa5689fe71271bb
I20260620 08:58:36.711198 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.702349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35417 (local address 127.0.98.62:36735)
0620 08:58:36.702506 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.702509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.703561 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 08:58:36.705395 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.705400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.705494 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.705581 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:36.706947 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.707686 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.708978 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.709205 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.709285 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.710489 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:36.710494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:36.710496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:36.710515 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:36.710548 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:36.710559 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:36.710660 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.711019 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.711080 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:58:36.712546 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.702726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36735 (local address 127.0.98.3:35417)
0620 08:58:36.703399 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.703440 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.705016 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.706009 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.706022 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.706148 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:36.706713 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.706734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.707822 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.707826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.708866 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.708878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.709347 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.709351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.709494 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.710115 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:36.710117 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:36.710157 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:36.710225 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:36.710766 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:36.710781 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:36.710789 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.710978 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.711356 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260620 08:58:36.713785 19502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36735
I20260620 08:58:36.714025 19502 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:36.714558 19502 heartbeater.cc:511] Master 127.0.98.62:36735 requested a full tablet report, sending...
I20260620 08:58:36.715651 19052 ts_manager.cc:194] Registered new tserver with Master: 448137f1ea7a48a2ab00e73fb0cf691d (127.0.98.3:42857)
I20260620 08:58:36.716224 19052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35417
I20260620 08:58:36.720270 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:58:36.726255 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.722349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.0.98.62:36735)
0620 08:58:36.722511 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.722515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.722533 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:58:36.722584 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.722587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.722637 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.722750 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:36.723127 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.723732 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.724609 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.724821 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.724902 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.725782 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:36.725785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:36.725787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:36.725801 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:36.725821 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:36.725829 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:36.725932 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.726104 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.726143 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:58:36.729070 19052 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:58:36.732048 19052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:58:36.746141 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.740888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35209 (local address 127.0.0.1:35510)
0620 08:58:36.741273 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.741317 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.741436 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.742144 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.742150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.742175 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:36.742310 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.742317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.743468 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.743472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.744420 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.744429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.745610 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.745613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.745735 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.745766 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.745856 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.745975 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260620 08:58:36.746608 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.741022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.0.98.2:35209)
0620 08:58:36.741441 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.741452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.741478 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:58:36.741563 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.741568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.741781 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.741933 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:36.742442 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.743384 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.744599 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.745373 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.745475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.745689 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.746340 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.746447 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":118,"threads_started":1}
I20260620 08:58:36.746697 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.740944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45695 (local address 127.0.0.1:59516)
0620 08:58:36.741865 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.741878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.741962 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.742649 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.742652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.742671 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:36.742755 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.742761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.744283 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.744286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.745476 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.745485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.746355 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.746359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.746444 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.746467 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.746509 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.746572 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":49,"spinlock_wait_cycles":1280,"thread_start_us":106,"threads_started":1}
I20260620 08:58:36.747318 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.741650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.0.98.1:45695)
0620 08:58:36.742207 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.742220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.742242 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:58:36.742314 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.742318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.742491 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.742894 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:36.743018 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.744164 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.745589 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.746221 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.746869 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.747069 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.747118 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.747192 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":103,"threads_started":1}
I20260620 08:58:36.747531 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.741629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42857 (local address 127.0.0.1:58632)
0620 08:58:36.742531 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.742554 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.742694 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.743165 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.743168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.743184 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:36.743269 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.743274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.744517 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.744520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.745766 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.745777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.747145 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.747148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.747237 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.747259 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.747333 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.747402 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":87,"thread_start_us":97,"threads_started":1}
I20260620 08:58:36.747990 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.742021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.0.98.3:42857)
0620 08:58:36.742387 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.742398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.742629 (+ 231us) server_negotiation.cc:407] Connection header received
0620 08:58:36.742844 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.742850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.743028 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.743176 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:36.743372 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.744415 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.745888 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.746830 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.746926 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.747085 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.747551 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.747808 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":88,"threads_started":1}
I20260620 08:58:36.749480 19305 tablet_service.cc:1467] Processing CreateTablet for tablet e0187cc3a0824636ae2a27a2e146b1fc (DEFAULT_TABLE table=test-table [id=6792395eea93470aa4059bb355765605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:36.749879 19174 tablet_service.cc:1467] Processing CreateTablet for tablet e0187cc3a0824636ae2a27a2e146b1fc (DEFAULT_TABLE table=test-table [id=6792395eea93470aa4059bb355765605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:36.750428 19305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0187cc3a0824636ae2a27a2e146b1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:36.750396 19436 tablet_service.cc:1467] Processing CreateTablet for tablet e0187cc3a0824636ae2a27a2e146b1fc (DEFAULT_TABLE table=test-table [id=6792395eea93470aa4059bb355765605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:36.750846 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0187cc3a0824636ae2a27a2e146b1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:36.751283 19436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0187cc3a0824636ae2a27a2e146b1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:36.756363 19519 tablet_bootstrap.cc:492] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Bootstrap starting.
I20260620 08:58:36.757743 19521 tablet_bootstrap.cc:492] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Bootstrap starting.
I20260620 08:58:36.759545 19521 tablet_bootstrap.cc:654] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:36.759555 19519 tablet_bootstrap.cc:654] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:36.759933 19520 tablet_bootstrap.cc:492] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: Bootstrap starting.
I20260620 08:58:36.760428 19521 log.cc:824] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:36.760428 19519 log.cc:824] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:36.762302 19521 tablet_bootstrap.cc:492] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: No bootstrap required, opened a new log
I20260620 08:58:36.762486 19521 ts_tablet_manager.cc:1397] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:58:36.762562 19520 tablet_bootstrap.cc:654] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:36.763338 19519 tablet_bootstrap.cc:492] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: No bootstrap required, opened a new log
I20260620 08:58:36.763355 19520 log.cc:824] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:36.763521 19519 ts_tablet_manager.cc:1397] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 08:58:36.765250 19520 tablet_bootstrap.cc:492] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: No bootstrap required, opened a new log
I20260620 08:58:36.765417 19520 ts_tablet_manager.cc:1397] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 08:58:36.766765 19521 raft_consensus.cc:348] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.767040 19521 raft_consensus.cc:374] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:36.767098 19521 raft_consensus.cc:729] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448137f1ea7a48a2ab00e73fb0cf691d, State: Initialized, Role: FOLLOWER
I20260620 08:58:36.767169 19519 raft_consensus.cc:348] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.767395 19519 raft_consensus.cc:374] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:36.767446 19519 raft_consensus.cc:729] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44988682bda4e53947eae7f578b6b7e, State: Initialized, Role: FOLLOWER
I20260620 08:58:36.767565 19521 consensus_queue.cc:260] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.767846 19519 consensus_queue.cc:260] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.768586 19521 ts_tablet_manager.cc:1428] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 08:58:36.768632 19519 ts_tablet_manager.cc:1428] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 08:58:36.768822 19371 heartbeater.cc:503] Master 127.0.98.62:36735 was elected leader, sending a full tablet report...
I20260620 08:58:36.768942 19502 heartbeater.cc:503] Master 127.0.98.62:36735 was elected leader, sending a full tablet report...
I20260620 08:58:36.770043 19520 raft_consensus.cc:348] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.770367 19520 raft_consensus.cc:374] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:36.770432 19520 raft_consensus.cc:729] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8859f243a86c49ffa7a7be73d01cabcb, State: Initialized, Role: FOLLOWER
I20260620 08:58:36.770828 19520 consensus_queue.cc:260] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.771756 19520 ts_tablet_manager.cc:1428] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 08:58:36.771857 19240 heartbeater.cc:503] Master 127.0.98.62:36735 was elected leader, sending a full tablet report...
W20260620 08:58:36.838382 19241 tablet.cc:2369] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:58:36.951514 19503 tablet.cc:2369] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:36.963404 19527 raft_consensus.cc:482] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:36.963654 19527 raft_consensus.cc:504] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.964756 19527 leader_election.cc:288] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d44988682bda4e53947eae7f578b6b7e (127.0.98.2:35209), 448137f1ea7a48a2ab00e73fb0cf691d (127.0.98.3:42857)
I20260620 08:58:36.968698 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.965144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40517 (local address 127.0.98.3:42857)
0620 08:58:36.965294 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.965299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.965483 (+ 184us) server_negotiation.cc:407] Connection header received
0620 08:58:36.965686 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.965689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.965746 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.965810 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:36.966164 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.966662 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.967792 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.968187 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.968282 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.968341 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.968535 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.968599 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:58:36.968693 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.965167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49491 (local address 127.0.98.2:35209)
0620 08:58:36.965297 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:36.965301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:36.965566 (+ 265us) server_negotiation.cc:407] Connection header received
0620 08:58:36.965669 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:36.965672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:36.965714 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:36.965783 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:36.966157 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.966659 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.967812 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.968173 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.968296 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.968357 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:36.968496 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 08:58:36.968565 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:58:36.968720 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.965041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35209 (local address 127.0.98.1:49491)
0620 08:58:36.965442 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.965454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.965581 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.965802 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.965805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.965820 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:36.966037 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.966043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.966765 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.966768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.967695 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.967705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.968282 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.968286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.968377 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.968407 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.968475 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.968544 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":87,"spinlock_wait_cycles":34688,"thread_start_us":76,"threads_started":1}
I20260620 08:58:36.968724 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:36.965041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42857 (local address 127.0.98.1:40517)
0620 08:58:36.965389 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:36.965401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:36.965510 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:36.965870 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:36.965873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:36.965890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:36.966053 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.966059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.966770 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.966774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.967687 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:36.967696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:36.968312 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:36.968316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:36.968419 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:36.968442 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:58:36.968513 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:58:36.968585 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":53,"thread_start_us":85,"threads_started":1}
I20260620 08:58:36.969483 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0187cc3a0824636ae2a27a2e146b1fc" candidate_uuid: "8859f243a86c49ffa7a7be73d01cabcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44988682bda4e53947eae7f578b6b7e" is_pre_election: true
I20260620 08:58:36.969489 19456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0187cc3a0824636ae2a27a2e146b1fc" candidate_uuid: "8859f243a86c49ffa7a7be73d01cabcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" is_pre_election: true
I20260620 08:58:36.969787 19325 raft_consensus.cc:2454] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8859f243a86c49ffa7a7be73d01cabcb in term 0.
I20260620 08:58:36.969787 19456 raft_consensus.cc:2454] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8859f243a86c49ffa7a7be73d01cabcb in term 0.
I20260620 08:58:36.970239 19130 leader_election.cc:302] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448137f1ea7a48a2ab00e73fb0cf691d, 8859f243a86c49ffa7a7be73d01cabcb; no voters:
I20260620 08:58:36.970484 19527 raft_consensus.cc:2781] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:58:36.970563 19527 raft_consensus.cc:482] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:58:36.970613 19527 raft_consensus.cc:3037] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:36.971982 19527 raft_consensus.cc:504] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.972338 19527 leader_election.cc:288] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [CANDIDATE]: Term 1 election: Requested vote from peers d44988682bda4e53947eae7f578b6b7e (127.0.98.2:35209), 448137f1ea7a48a2ab00e73fb0cf691d (127.0.98.3:42857)
I20260620 08:58:36.972656 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0187cc3a0824636ae2a27a2e146b1fc" candidate_uuid: "8859f243a86c49ffa7a7be73d01cabcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44988682bda4e53947eae7f578b6b7e"
I20260620 08:58:36.972695 19456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0187cc3a0824636ae2a27a2e146b1fc" candidate_uuid: "8859f243a86c49ffa7a7be73d01cabcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448137f1ea7a48a2ab00e73fb0cf691d"
I20260620 08:58:36.972781 19325 raft_consensus.cc:3037] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:36.972813 19456 raft_consensus.cc:3037] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:36.973829 19325 raft_consensus.cc:2454] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8859f243a86c49ffa7a7be73d01cabcb in term 1.
I20260620 08:58:36.973838 19456 raft_consensus.cc:2454] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8859f243a86c49ffa7a7be73d01cabcb in term 1.
I20260620 08:58:36.974133 19130 leader_election.cc:302] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448137f1ea7a48a2ab00e73fb0cf691d, 8859f243a86c49ffa7a7be73d01cabcb; no voters:
I20260620 08:58:36.974295 19527 raft_consensus.cc:2781] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:36.974529 19527 raft_consensus.cc:686] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 1 LEADER]: Becoming Leader. State: Replica: 8859f243a86c49ffa7a7be73d01cabcb, State: Running, Role: LEADER
I20260620 08:58:36.974854 19527 consensus_queue.cc:237] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } }
I20260620 08:58:36.978421 19050 catalog_manager.cc:5495] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8859f243a86c49ffa7a7be73d01cabcb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8859f243a86c49ffa7a7be73d01cabcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8859f243a86c49ffa7a7be73d01cabcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 } health_report { overall_health: UNKNOWN } } }
W20260620 08:58:37.016080 19372 tablet.cc:2369] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:37.041913 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:37.036999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.0.98.1:45695)
0620 08:58:37.037115 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:37.037118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:37.037557 (+ 439us) server_negotiation.cc:407] Connection header received
0620 08:58:37.037720 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:37.037723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:37.037795 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:37.037903 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:58:37.038209 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:37.038930 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:37.040214 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:37.040445 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:37.040546 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:37.040872 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:58:37.041403 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:58:37.041563 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:37.041701 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 08:58:37.041743 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:58:37.049331 19456 raft_consensus.cc:1261] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 1 FOLLOWER]: Refusing update from remote peer 8859f243a86c49ffa7a7be73d01cabcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:37.050038 19527 consensus_queue.cc:1035] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "448137f1ea7a48a2ab00e73fb0cf691d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:37.050251 19325 raft_consensus.cc:1261] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 1 FOLLOWER]: Refusing update from remote peer 8859f243a86c49ffa7a7be73d01cabcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:37.050911 19527 consensus_queue.cc:1035] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44988682bda4e53947eae7f578b6b7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:37.056823 19533 mvcc.cc:205] Tried to move back new op lower bound from 7298850476225400832 to 7298850475931471872. Current Snapshot: MvccSnapshot[applied={T|T < 7298850476225400832}]
I20260620 08:58:37.068041 19051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:37.068222 19051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:37.071583 19534 mvcc.cc:205] Tried to move back new op lower bound from 7298850476225400832 to 7298850475931471872. Current Snapshot: MvccSnapshot[applied={T|T < 7298850476225400832}]
I20260620 08:58:37.072717 19051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf12a2288f0d48bc85e2e25093c0bb05: Sending DeleteTablet for 3 replicas of tablet e0187cc3a0824636ae2a27a2e146b1fc
I20260620 08:58:37.073426 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet e0187cc3a0824636ae2a27a2e146b1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:37 UTC) from {username='slave'} at 127.0.0.1:59516
I20260620 08:58:37.073424 19436 tablet_service.cc:1514] Processing DeleteTablet for tablet e0187cc3a0824636ae2a27a2e146b1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:37 UTC) from {username='slave'} at 127.0.0.1:58632
I20260620 08:58:37.073437 19305 tablet_service.cc:1514] Processing DeleteTablet for tablet e0187cc3a0824636ae2a27a2e146b1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:37 UTC) from {username='slave'} at 127.0.0.1:35510
I20260620 08:58:37.073894 19543 tablet_replica.cc:330] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: stopping tablet replica
I20260620 08:58:37.074299 19543 raft_consensus.cc:2229] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:37.074585 19543 raft_consensus.cc:2258] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:37.074916 19542 tablet_replica.cc:330] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb: stopping tablet replica
I20260620 08:58:37.075178 19542 raft_consensus.cc:2229] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:58:37.075208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19112
I20260620 08:58:37.075452 19542 raft_consensus.cc:2258] T e0187cc3a0824636ae2a27a2e146b1fc P 8859f243a86c49ffa7a7be73d01cabcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:37.076100 19544 tablet_replica.cc:330] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: stopping tablet replica
I20260620 08:58:37.076365 19544 raft_consensus.cc:2229] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:37.076656 19544 raft_consensus.cc:2258] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:37.077589 19543 ts_tablet_manager.cc:1905] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:37.079699 19543 ts_tablet_manager.cc:1918] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:37.079838 19543 log.cc:1196] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/e0187cc3a0824636ae2a27a2e146b1fc
I20260620 08:58:37.080192 19543 ts_tablet_manager.cc:1939] T e0187cc3a0824636ae2a27a2e146b1fc P 448137f1ea7a48a2ab00e73fb0cf691d: Deleting consensus metadata
I20260620 08:58:37.080588 19544 ts_tablet_manager.cc:1905] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:37.081466 19039 catalog_manager.cc:4841] TS 448137f1ea7a48a2ab00e73fb0cf691d (127.0.98.3:42857): tablet e0187cc3a0824636ae2a27a2e146b1fc (table test-table [id=6792395eea93470aa4059bb355765605]) successfully deleted
I20260620 08:58:37.082768 19544 ts_tablet_manager.cc:1918] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:37.082851 19544 log.cc:1196] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/e0187cc3a0824636ae2a27a2e146b1fc
I20260620 08:58:37.083099 19544 ts_tablet_manager.cc:1939] T e0187cc3a0824636ae2a27a2e146b1fc P d44988682bda4e53947eae7f578b6b7e: Deleting consensus metadata
W20260620 08:58:37.083379 19036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45695 (error 108)
I20260620 08:58:37.084165 19038 catalog_manager.cc:4841] TS d44988682bda4e53947eae7f578b6b7e (127.0.98.2:35209): tablet e0187cc3a0824636ae2a27a2e146b1fc (table test-table [id=6792395eea93470aa4059bb355765605]) successfully deleted
I20260620 08:58:37.084297 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:37.084036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45695 (local address 127.0.0.1:59534)
0620 08:58:37.084149 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:37.084219 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 08:58:37.084379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19243
W20260620 08:58:37.091717 19036 catalog_manager.cc:4584] TS 8859f243a86c49ffa7a7be73d01cabcb (127.0.98.1:45695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0187cc3a0824636ae2a27a2e146b1fc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45695: connect: Connection refused (error 111)
I20260620 08:58:37.092245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19374
I20260620 08:58:37.098333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19021
2026-06-20T08:58:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1178 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:37 dist-test-slave-fb1v krb5kdc[19548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:37 dist-test-slave-fb1v krb5kdc[19548](info): set up 2 sockets
Jun 20 08:58:37 dist-test-slave-fb1v krb5kdc[19548](info): commencing operation
krb5kdc: starting...
W20260620 08:58:38.047546 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:58:38.163826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:38.271366 19561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:38.271687 19561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:38.271747 19561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:38.275787 19561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:38.275849 19561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:38.275876 19561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:38.275897 19561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:38.275920 19561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:38.281396 19561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:38.282642 19561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:38.283864 19561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:38.292492 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:38.292492 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:38.292493 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.292534 19561 server_base.cc:1034] running on GCE node
I20260620 08:58:38.293917 19561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:38.295116 19561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:38.296289 19561 hybrid_clock.cc:648] HybridClock initialized: now 1781945918296272 us; error 36 us; skew 500 ppm
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:38.298969 19561 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:58:38.299887 19561 webserver.cc:460] Webserver started at http://127.0.98.62:42261/ using document root <none> and password file <none>
I20260620 08:58:38.300411 19561 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:38.300477 19561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:38.300683 19561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:38.302690 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "a1cebba8ea474f18b510b88bd0fd6bea"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "b5084db07e2a91db4bfca78925d05f69"
server_key_iv: "c21a114cfb235c4a387fbbbb8940f78a"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.303205 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1cebba8ea474f18b510b88bd0fd6bea"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "b5084db07e2a91db4bfca78925d05f69"
server_key_iv: "c21a114cfb235c4a387fbbbb8940f78a"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.306756 19561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:58:38.309043 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.310241 19561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:58:38.310359 19561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "a1cebba8ea474f18b510b88bd0fd6bea"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "b5084db07e2a91db4bfca78925d05f69"
server_key_iv: "c21a114cfb235c4a387fbbbb8940f78a"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.310458 19561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:38.324234 19561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:38.327234 19561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:38.327416 19561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:38.334106 19561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37619
I20260620 08:58:38.334148 19628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37619 every 8 connection(s)
I20260620 08:58:38.335019 19561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:38.337690 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:38.340229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19561
I20260620 08:58:38.340342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:38.340603 392 external_mini_cluster.cc:1428] Setting key 9f22679a5400bbf161d68da30ffa7543
I20260620 08:58:38.343196 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea: Bootstrap starting.
I20260620 08:58:38.345600 19629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:38.346446 19629 log.cc:824] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:38.348742 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea: No bootstrap required, opened a new log
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:38.351984 19629 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } }
I20260620 08:58:38.352195 19629 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:38.352227 19629 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cebba8ea474f18b510b88bd0fd6bea, State: Initialized, Role: FOLLOWER
I20260620 08:58:38.352802 19629 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } }
I20260620 08:58:38.352948 19629 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:38.353016 19629 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:38.353112 19629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:38.354166 19629 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } }
I20260620 08:58:38.354550 19629 leader_election.cc:302] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cebba8ea474f18b510b88bd0fd6bea; no voters:
I20260620 08:58:38.355206 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:38.341813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.0.98.62:37619)
0620 08:58:38.342202 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:38.342217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:38.342251 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:58:38.342852 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:38.342867 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:38.343175 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:38.343451 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:38.344310 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.345325 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.346454 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.346959 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.347096 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:38.350174 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:38.350179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:38.350192 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:38.350219 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:38.352477 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:38.352880 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:38.352886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:38.352892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:38.353030 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:38.353349 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:38.353352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:38.353353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:38.353729 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:38.353874 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:38.354185 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 08:58:38.354445 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260620 08:58:38.355326 19629 leader_election.cc:288] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:38.355449 19634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:38.355695 19634 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [term 1 LEADER]: Becoming Leader. State: Replica: a1cebba8ea474f18b510b88bd0fd6bea, State: Running, Role: LEADER
I20260620 08:58:38.356115 19634 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } }
I20260620 08:58:38.356983 19629 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:38.357909 19636 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } }
I20260620 08:58:38.358059 19636 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:38.358431 19643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:38.358393 19635 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cebba8ea474f18b510b88bd0fd6bea. Latest consensus state: current_term: 1 leader_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cebba8ea474f18b510b88bd0fd6bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } }
I20260620 08:58:38.358961 19635 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:38.361194 19643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:38.366272 19643 catalog_manager.cc:1347] Generated new cluster ID: 87dfd1996e584e5ea027355b969868be
I20260620 08:58:38.366338 19643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:38.397503 19643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:38.398432 19643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:38.407470 19643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea: Generated new TSK 0
I20260620 08:58:38.408236 19643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:58:38.438951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37619
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:38.544402 19655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:38.544653 19655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:38.544694 19655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:38.548583 19655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:38.548636 19655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:38.548691 19655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:38.553804 19655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:38.554756 19655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:38.555703 19655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:38.565018 19660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.565187 19655 server_base.cc:1034] running on GCE node
W20260620 08:58:38.565245 19663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:38.565137 19661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.565652 19655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:38.566218 19655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:38.567378 19655 hybrid_clock.cc:648] HybridClock initialized: now 1781945918567363 us; error 28 us; skew 500 ppm
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:38.569954 19655 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:58:38.570729 19655 webserver.cc:460] Webserver started at http://127.0.98.1:32931/ using document root <none> and password file <none>
I20260620 08:58:38.571214 19655 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:38.571267 19655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:38.571424 19655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:38.573328 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "03008c8eb63145aab0abe19899c0188a"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "08520ece3ea59f924f71252523fb27e2"
server_key_iv: "4a3819f1ddb299afecfc1c978aa5333b"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.573835 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03008c8eb63145aab0abe19899c0188a"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "08520ece3ea59f924f71252523fb27e2"
server_key_iv: "4a3819f1ddb299afecfc1c978aa5333b"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.577248 19655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:38.579527 19670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.580718 19655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:38.580811 19655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "03008c8eb63145aab0abe19899c0188a"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "08520ece3ea59f924f71252523fb27e2"
server_key_iv: "4a3819f1ddb299afecfc1c978aa5333b"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.580891 19655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:38.596659 19655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:38.599658 19655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:38.599804 19655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:38.600323 19655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:38.601071 19655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:38.601123 19655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.601161 19655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:38.601176 19655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.610076 19655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32987
I20260620 08:58:38.610158 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32987 every 8 connection(s)
I20260620 08:58:38.610932 19655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:38.614459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19655
I20260620 08:58:38.614586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:58:38.614890 392 external_mini_cluster.cc:1428] Setting key 227824e4148fb5b8655b0f0f09d10dc8
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:58:38.626315 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:38.613411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43197 (local address 127.0.98.62:37619)
0620 08:58:38.613552 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:38.613555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:38.614603 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 08:58:38.615692 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:38.615695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:38.615747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:38.615822 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:38.617210 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.617875 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.619330 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.619690 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.619794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:38.622165 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:38.622168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:38.622171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:38.622199 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:38.623857 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:38.624391 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:38.624394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:38.624395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:38.624470 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:38.624981 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:38.624986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:38.624988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:38.625304 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:38.625440 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:38.626073 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 08:58:38.626187 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:58:38.627103 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:38.613822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37619 (local address 127.0.98.1:43197)
0620 08:58:38.614386 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:38.614446 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:38.615487 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:38.616000 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:38.616007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:38.616408 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:38.617050 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:38.617069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.618019 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.618023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:38.619198 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:38.619208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.619853 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.619864 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:38.620007 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:38.620604 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:38.620620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:38.621999 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:38.623996 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:38.624003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:38.624018 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:38.624268 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:38.624596 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:38.624599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:38.624600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:38.624738 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:38.625480 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:38.625486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:38.625736 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 08:58:38.625954 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 08:58:38.626236 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":119,"threads_started":1}
I20260620 08:58:38.628374 19784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37619
I20260620 08:58:38.628618 19784 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:38.629077 19784 heartbeater.cc:511] Master 127.0.98.62:37619 requested a full tablet report, sending...
I20260620 08:58:38.630467 19593 ts_manager.cc:194] Registered new tserver with Master: 03008c8eb63145aab0abe19899c0188a (127.0.98.1:32987)
I20260620 08:58:38.631561 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:58:38.644955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37619
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:38.749485 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:38.749727 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:38.749768 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:38.753739 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:38.753800 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:38.753857 19789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:38.759123 19789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:38.760100 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:38.761053 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:38.770334 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.770490 19789 server_base.cc:1034] running on GCE node
W20260620 08:58:38.770334 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:38.770407 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.770960 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:38.771638 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:38.772828 19789 hybrid_clock.cc:648] HybridClock initialized: now 1781945918772800 us; error 38 us; skew 500 ppm
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:38.775547 19789 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:58:38.776444 19789 webserver.cc:460] Webserver started at http://127.0.98.2:46065/ using document root <none> and password file <none>
I20260620 08:58:38.776958 19789 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:38.777024 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:38.777221 19789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:38.779153 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c73bd7aa0064e8e9b8989837cb693b6"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "b9b7c2a84d3d6e9892729b457c691ebd"
server_key_iv: "295ad303e8e213eedbb0d58aea3d3542"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.779706 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c73bd7aa0064e8e9b8989837cb693b6"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "b9b7c2a84d3d6e9892729b457c691ebd"
server_key_iv: "295ad303e8e213eedbb0d58aea3d3542"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.783157 19789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:38.785698 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.786880 19789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:38.786998 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "2c73bd7aa0064e8e9b8989837cb693b6"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "b9b7c2a84d3d6e9892729b457c691ebd"
server_key_iv: "295ad303e8e213eedbb0d58aea3d3542"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.787109 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:38.800204 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:38.803028 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:38.803221 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:38.803822 19789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:38.804672 19789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:38.804720 19789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.804759 19789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:38.804775 19789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.813341 19789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39941
I20260620 08:58:38.813366 19917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39941 every 8 connection(s)
I20260620 08:58:38.814244 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:38.820781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19789
I20260620 08:58:38.820906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:58:38.821148 392 external_mini_cluster.cc:1428] Setting key 939de882671744b2b858b16f56433497
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:38.828253 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:38.816517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48801 (local address 127.0.98.62:37619)
0620 08:58:38.816706 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:38.816710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:38.817961 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 08:58:38.819156 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:38.819159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:38.819210 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:38.819298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:38.820563 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.821172 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.822386 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.822684 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.822794 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:38.825124 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:38.825127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:38.825129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:38.825150 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:38.826368 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:38.826818 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:38.826821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:38.826823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:38.826890 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:38.827189 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:38.827192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:38.827194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:38.827364 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:38.827447 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:38.827978 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 08:58:38.828094 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 08:58:38.829043 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:38.817273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37619 (local address 127.0.98.2:48801)
0620 08:58:38.817765 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:38.817814 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:38.818963 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:38.819496 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:38.819504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:38.819915 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:38.820420 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:38.820433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.821307 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.821310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:38.822278 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:38.822290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:38.822856 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:38.822859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:38.823009 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:38.823584 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:38.823599 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:38.824971 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:38.826474 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:38.826479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:38.826494 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:38.826729 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:38.826974 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:38.826977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:38.826979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:38.827106 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:38.827463 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:38.827470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:38.827749 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 08:58:38.827914 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 08:58:38.828223 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:58:38.830283 19918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37619
I20260620 08:58:38.830515 19918 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:38.830991 19918 heartbeater.cc:511] Master 127.0.98.62:37619 requested a full tablet report, sending...
I20260620 08:58:38.831996 19593 ts_manager.cc:194] Registered new tserver with Master: 2c73bd7aa0064e8e9b8989837cb693b6 (127.0.98.2:39941)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:58:38.832576 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:58:38.846967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37619
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:38.952759 19923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:38.953011 19923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:38.953068 19923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:38.956979 19923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:38.957041 19923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:38.957119 19923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:38.962364 19923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:38.963318 19923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:38.964280 19923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:38.973655 19929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.973722 19923 server_base.cc:1034] running on GCE node
W20260620 08:58:38.973663 19928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:38.973675 19931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:38.974282 19923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:38.974864 19923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:38.976035 19923 hybrid_clock.cc:648] HybridClock initialized: now 1781945918976010 us; error 38 us; skew 500 ppm
Jun 20 08:58:38 dist-test-slave-fb1v krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:38.978596 19923 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:58:38.979372 19923 webserver.cc:460] Webserver started at http://127.0.98.3:45245/ using document root <none> and password file <none>
I20260620 08:58:38.979931 19923 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:38.979992 19923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:38.980146 19923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:38.982084 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "2991671121b9427aae2acc3168558948"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "df642c90141bc567bf2c8b7aa0c12c3d"
server_key_iv: "ae2d77148d52093370c4ad5e7d588421"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.982574 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2991671121b9427aae2acc3168558948"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "df642c90141bc567bf2c8b7aa0c12c3d"
server_key_iv: "ae2d77148d52093370c4ad5e7d588421"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.986094 19923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 08:58:38.988406 19938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:38.989609 19923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 08:58:38.989711 19923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "2991671121b9427aae2acc3168558948"
format_stamp: "Formatted at 2026-06-20 08:58:38 on dist-test-slave-fb1v"
server_key: "df642c90141bc567bf2c8b7aa0c12c3d"
server_key_iv: "ae2d77148d52093370c4ad5e7d588421"
server_key_version: "encryptionkey@0"
I20260620 08:58:38.989794 19923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:39.006914 19923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:39.010274 19923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:39.010434 19923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:39.010977 19923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:39.011796 19923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:39.011842 19923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.011881 19923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:39.011895 19923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.021045 19923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35815
I20260620 08:58:39.021061 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35815 every 8 connection(s)
I20260620 08:58:39.021986 19923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:39.022782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 19923
I20260620 08:58:39.022890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:58:39.023161 392 external_mini_cluster.cc:1428] Setting key f54e06ba3e31ef4d9506a1508aeb0617
Jun 20 08:58:39 dist-test-slave-fb1v krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:39.036162 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.024230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35427 (local address 127.0.98.62:37619)
0620 08:58:39.024352 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.024356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.025552 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 08:58:39.026857 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.026860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.026906 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.026979 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:39.028157 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.028716 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.029896 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.030123 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.030212 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.032676 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:39.032679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:39.032681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:39.032704 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:39.034165 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:39.034654 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:39.034657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:39.034659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:39.034730 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:39.035034 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:39.035038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:39.035039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:39.035206 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:39.035289 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.035803 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.035920 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:58:39.037045 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.024844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37619 (local address 127.0.98.3:35427)
0620 08:58:39.025405 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.025460 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.026649 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.027119 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.027125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.027472 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:39.028008 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.028021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.028841 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.028844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.029768 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.029778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.030285 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.030288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.030437 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.031008 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:58:39.031024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:39.032498 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:39.034284 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:39.034289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:39.034303 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:39.034562 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:39.034809 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:58:39.034812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:58:39.034814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:58:39.034941 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:58:39.035301 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:39.035307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:39.035540 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.035740 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.036138 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":127,"threads_started":1}
I20260620 08:58:39.038017 20052 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37619
I20260620 08:58:39.038236 20052 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:39.038684 20052 heartbeater.cc:511] Master 127.0.98.62:37619 requested a full tablet report, sending...
I20260620 08:58:39.039669 19593 ts_manager.cc:194] Registered new tserver with Master: 2991671121b9427aae2acc3168558948 (127.0.98.3:35815)
I20260620 08:58:39.040221 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35427
I20260620 08:58:39.049032 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:58:39.057886 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.050818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.0.98.62:37619)
0620 08:58:39.050939 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.050943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.051102 (+ 159us) server_negotiation.cc:407] Connection header received
0620 08:58:39.051170 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.051172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.051213 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.051284 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:39.051962 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.052465 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.053241 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.053460 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.053549 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.054798 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:39.054801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:39.054803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:39.054824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:39.056093 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:39.056546 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:39.056549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:39.056551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:39.056637 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:39.056989 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:39.056993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:39.056994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:39.057266 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:39.057382 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.057615 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.057736 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:58:39.060794 19593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:58:39.063812 19593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:58:39.077610 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.073092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.0.98.1:32987)
0620 08:58:39.073454 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.073461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.073478 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:58:39.073537 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.073541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.073689 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.074050 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:39.074179 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.075224 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.076447 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.076968 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.077064 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.077197 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.077372 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.077467 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":102,"threads_started":1}
I20260620 08:58:39.077603 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.072663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32987 (local address 127.0.0.1:36782)
0620 08:58:39.073092 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.073104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.073175 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.073816 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.073818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.073831 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:39.073926 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.073931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.075358 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.075361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.076340 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.076347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.077076 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.077080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.077194 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.077232 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.077330 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.077456 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":80,"threads_started":1}
I20260620 08:58:39.077903 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.072529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.0.98.3:35815)
0620 08:58:39.073388 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.073410 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.073432 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:58:39.073500 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.073505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.073670 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.073804 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:39.074002 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.074822 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.076114 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.076871 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.076967 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.077095 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.077666 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.077758 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":112,"threads_started":1}
I20260620 08:58:39.077903 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.071861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.0.98.2:39941)
0620 08:58:39.072157 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.072173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.072196 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:58:39.072340 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.072345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.072561 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.072712 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:39.073157 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.074255 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.076039 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.076871 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.076952 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.077093 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.077527 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.077638 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20260620 08:58:39.077970 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.072381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35815 (local address 127.0.0.1:53350)
0620 08:58:39.072598 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.072610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.072688 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.073798 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.073802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.073819 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:39.073914 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.073920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.074945 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.074948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.075995 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.076005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.077157 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.077161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.077354 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.077402 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.077466 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.077537 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":69,"spinlock_wait_cycles":8576,"thread_start_us":89,"threads_started":1}
I20260620 08:58:39.078336 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.071688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39941 (local address 127.0.0.1:49770)
0620 08:58:39.072014 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.072061 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.072182 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.072843 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.072849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.072872 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:39.073007 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.073014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.074384 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.074388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.075856 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.075864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.077174 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.077177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.077350 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.077402 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.078176 (+ 774us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.078248 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":58,"thread_start_us":89,"threads_started":1}
I20260620 08:58:39.080525 19718 tablet_service.cc:1467] Processing CreateTablet for tablet 12abe3dbc7d446bca956573a641e2b55 (DEFAULT_TABLE table=test-table [id=faa2f6a2e6df416980cccd4b47b8d8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:39.080869 19986 tablet_service.cc:1467] Processing CreateTablet for tablet 12abe3dbc7d446bca956573a641e2b55 (DEFAULT_TABLE table=test-table [id=faa2f6a2e6df416980cccd4b47b8d8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:39.080869 19852 tablet_service.cc:1467] Processing CreateTablet for tablet 12abe3dbc7d446bca956573a641e2b55 (DEFAULT_TABLE table=test-table [id=faa2f6a2e6df416980cccd4b47b8d8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:39.081472 19718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12abe3dbc7d446bca956573a641e2b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:39.081816 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12abe3dbc7d446bca956573a641e2b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:39.081816 19852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12abe3dbc7d446bca956573a641e2b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:39.087930 20069 tablet_bootstrap.cc:492] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: Bootstrap starting.
I20260620 08:58:39.089193 20071 tablet_bootstrap.cc:492] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Bootstrap starting.
I20260620 08:58:39.090098 20069 tablet_bootstrap.cc:654] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:39.090867 20071 tablet_bootstrap.cc:654] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:39.091097 20069 log.cc:824] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:39.091868 20071 log.cc:824] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:39.092474 20070 tablet_bootstrap.cc:492] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Bootstrap starting.
I20260620 08:58:39.092947 20069 tablet_bootstrap.cc:492] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: No bootstrap required, opened a new log
I20260620 08:58:39.093140 20069 ts_tablet_manager.cc:1397] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 08:58:39.094556 20070 tablet_bootstrap.cc:654] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:39.095157 20070 log.cc:824] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:39.096050 20069 raft_consensus.cc:348] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.096254 20069 raft_consensus.cc:374] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:39.096287 20069 raft_consensus.cc:729] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03008c8eb63145aab0abe19899c0188a, State: Initialized, Role: FOLLOWER
I20260620 08:58:39.096597 20071 tablet_bootstrap.cc:492] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: No bootstrap required, opened a new log
I20260620 08:58:39.096685 20069 consensus_queue.cc:260] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.096758 20071 ts_tablet_manager.cc:1397] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 08:58:39.097304 19784 heartbeater.cc:503] Master 127.0.98.62:37619 was elected leader, sending a full tablet report...
I20260620 08:58:39.097571 20069 ts_tablet_manager.cc:1428] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260620 08:58:39.099782 20071 raft_consensus.cc:348] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.100044 20071 raft_consensus.cc:374] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:39.100087 20071 raft_consensus.cc:729] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2991671121b9427aae2acc3168558948, State: Initialized, Role: FOLLOWER
I20260620 08:58:39.100503 20071 consensus_queue.cc:260] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.100875 20070 tablet_bootstrap.cc:492] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: No bootstrap required, opened a new log
I20260620 08:58:39.101049 20070 ts_tablet_manager.cc:1397] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 08:58:39.101105 20052 heartbeater.cc:503] Master 127.0.98.62:37619 was elected leader, sending a full tablet report...
I20260620 08:58:39.101433 20071 ts_tablet_manager.cc:1428] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:58:39.103856 20070 raft_consensus.cc:348] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.104066 20070 raft_consensus.cc:374] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:39.104099 20070 raft_consensus.cc:729] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c73bd7aa0064e8e9b8989837cb693b6, State: Initialized, Role: FOLLOWER
I20260620 08:58:39.104441 20070 consensus_queue.cc:260] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.104998 19918 heartbeater.cc:503] Master 127.0.98.62:37619 was elected leader, sending a full tablet report...
I20260620 08:58:39.105227 20070 ts_tablet_manager.cc:1428] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260620 08:58:39.112143 19785 tablet.cc:2369] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:39.165864 20076 raft_consensus.cc:482] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:39.166126 20076 raft_consensus.cc:504] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.167359 20076 leader_election.cc:288] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c73bd7aa0064e8e9b8989837cb693b6 (127.0.98.2:39941), 03008c8eb63145aab0abe19899c0188a (127.0.98.1:32987)
I20260620 08:58:39.171307 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.167553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39941 (local address 127.0.98.3:48643)
0620 08:58:39.167707 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.167723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.167808 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.168135 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.168138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.168159 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:39.168265 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.168272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.169110 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.169114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.170153 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.170164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.170913 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.170916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.171020 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.171046 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.171113 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.171181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 08:58:39.171319 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.167683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48643 (local address 127.0.98.2:39941)
0620 08:58:39.167795 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.167800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.167817 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:58:39.167898 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.167902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.167965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.168051 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:39.168373 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.168998 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.170276 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.170810 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.170886 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.170937 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.171127 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.171197 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:58:39.171514 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.167989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34641 (local address 127.0.98.1:32987)
0620 08:58:39.168120 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.168124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.168261 (+ 137us) server_negotiation.cc:407] Connection header received
0620 08:58:39.168349 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.168352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.168396 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.168449 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:39.168755 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.169344 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.170557 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.171007 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.171108 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.171164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.171338 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.171409 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:58:39.171517 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.167846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32987 (local address 127.0.98.3:34641)
0620 08:58:39.168186 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.168197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.168276 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.168477 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.168480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.168495 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:39.168622 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.168628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.169483 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.169486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.170440 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.170450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.171122 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.171126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.171221 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.171242 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.171315 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.171376 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":136,"threads_started":1}
I20260620 08:58:39.172065 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abe3dbc7d446bca956573a641e2b55" candidate_uuid: "2991671121b9427aae2acc3168558948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" is_pre_election: true
I20260620 08:58:39.172189 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abe3dbc7d446bca956573a641e2b55" candidate_uuid: "2991671121b9427aae2acc3168558948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03008c8eb63145aab0abe19899c0188a" is_pre_election: true
I20260620 08:58:39.172391 19872 raft_consensus.cc:2454] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2991671121b9427aae2acc3168558948 in term 0.
I20260620 08:58:39.172418 19738 raft_consensus.cc:2454] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2991671121b9427aae2acc3168558948 in term 0.
I20260620 08:58:39.172832 19940 leader_election.cc:302] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03008c8eb63145aab0abe19899c0188a, 2991671121b9427aae2acc3168558948; no voters:
I20260620 08:58:39.173060 20076 raft_consensus.cc:2781] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:58:39.173141 20076 raft_consensus.cc:482] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:58:39.173189 20076 raft_consensus.cc:3037] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:39.174163 20076 raft_consensus.cc:504] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.174453 20076 leader_election.cc:288] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [CANDIDATE]: Term 1 election: Requested vote from peers 2c73bd7aa0064e8e9b8989837cb693b6 (127.0.98.2:39941), 03008c8eb63145aab0abe19899c0188a (127.0.98.1:32987)
I20260620 08:58:39.174791 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abe3dbc7d446bca956573a641e2b55" candidate_uuid: "2991671121b9427aae2acc3168558948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c73bd7aa0064e8e9b8989837cb693b6"
I20260620 08:58:39.174841 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abe3dbc7d446bca956573a641e2b55" candidate_uuid: "2991671121b9427aae2acc3168558948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03008c8eb63145aab0abe19899c0188a"
I20260620 08:58:39.174911 19872 raft_consensus.cc:3037] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:39.174940 19738 raft_consensus.cc:3037] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:39.175881 19738 raft_consensus.cc:2454] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2991671121b9427aae2acc3168558948 in term 1.
I20260620 08:58:39.175988 19872 raft_consensus.cc:2454] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2991671121b9427aae2acc3168558948 in term 1.
I20260620 08:58:39.176136 19940 leader_election.cc:302] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03008c8eb63145aab0abe19899c0188a, 2991671121b9427aae2acc3168558948; no voters:
I20260620 08:58:39.176290 20076 raft_consensus.cc:2781] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:39.176522 20076 raft_consensus.cc:686] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 1 LEADER]: Becoming Leader. State: Replica: 2991671121b9427aae2acc3168558948, State: Running, Role: LEADER
I20260620 08:58:39.176849 20076 consensus_queue.cc:237] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260620 08:58:39.179831 19592 catalog_manager.cc:5495] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2991671121b9427aae2acc3168558948 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2991671121b9427aae2acc3168558948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2991671121b9427aae2acc3168558948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } health_report { overall_health: UNKNOWN } } }
I20260620 08:58:39.198774 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.195698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.98.3:35815)
0620 08:58:39.195811 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.195814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.195827 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:58:39.195898 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.195901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.195953 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.196037 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:58:39.196226 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.196712 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.197555 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.197762 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.197836 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.198101 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:58:39.198489 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:58:39.198574 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.198600 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.198647 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:58:39.204278 19738 raft_consensus.cc:1261] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 1 FOLLOWER]: Refusing update from remote peer 2991671121b9427aae2acc3168558948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:39.204278 19872 raft_consensus.cc:1261] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 1 FOLLOWER]: Refusing update from remote peer 2991671121b9427aae2acc3168558948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:39.204977 20076 consensus_queue.cc:1035] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c73bd7aa0064e8e9b8989837cb693b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:39.205179 20079 consensus_queue.cc:1035] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03008c8eb63145aab0abe19899c0188a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:39.212529 20084 mvcc.cc:205] Tried to move back new op lower bound from 7298850485053341696 to 7298850484950720512. Current Snapshot: MvccSnapshot[applied={T|T < 7298850485053341696}]
I20260620 08:58:39.220520 19591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:39.220674 19591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:39.222927 19591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1cebba8ea474f18b510b88bd0fd6bea: Sending DeleteTablet for 3 replicas of tablet 12abe3dbc7d446bca956573a641e2b55
I20260620 08:58:39.223594 19852 tablet_service.cc:1514] Processing DeleteTablet for tablet 12abe3dbc7d446bca956573a641e2b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49770
I20260620 08:58:39.223692 19986 tablet_service.cc:1514] Processing DeleteTablet for tablet 12abe3dbc7d446bca956573a641e2b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53350
I20260620 08:58:39.224110 20092 tablet_replica.cc:330] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: stopping tablet replica
I20260620 08:58:39.224123 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet 12abe3dbc7d446bca956573a641e2b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36782
I20260620 08:58:39.224531 20094 tablet_replica.cc:330] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a: stopping tablet replica
I20260620 08:58:39.224524 20092 raft_consensus.cc:2229] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:39.224803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19655
I20260620 08:58:39.224810 20094 raft_consensus.cc:2229] T 12abe3dbc7d446bca956573a641e2b55 P 03008c8eb63145aab0abe19899c0188a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:39.225322 20092 raft_consensus.cc:2258] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:39.227066 20092 ts_tablet_manager.cc:1905] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:39.229873 20092 ts_tablet_manager.cc:1918] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:39.229971 20092 log.cc:1196] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/12abe3dbc7d446bca956573a641e2b55
I20260620 08:58:39.230283 20092 ts_tablet_manager.cc:1939] T 12abe3dbc7d446bca956573a641e2b55 P 2c73bd7aa0064e8e9b8989837cb693b6: Deleting consensus metadata
I20260620 08:58:39.230268 20093 tablet_replica.cc:330] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: stopping tablet replica
I20260620 08:58:39.230645 20093 raft_consensus.cc:2229] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:58:39.231109 20093 raft_consensus.cc:2258] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 08:58:39.231511 19578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32987 (error 108)
I20260620 08:58:39.231998 19578 catalog_manager.cc:4841] TS 2c73bd7aa0064e8e9b8989837cb693b6 (127.0.98.2:39941): tablet 12abe3dbc7d446bca956573a641e2b55 (table test-table [id=faa2f6a2e6df416980cccd4b47b8d8ec]) successfully deleted
W20260620 08:58:39.232553 20063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:58:39.232286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32987 (local address 127.0.0.1:36790)
0620 08:58:39.232409 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.232469 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32987: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:58:39.232544 20093 ts_tablet_manager.cc:1905] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:58:39.232666 19578 catalog_manager.cc:4584] TS 03008c8eb63145aab0abe19899c0188a (127.0.98.1:32987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12abe3dbc7d446bca956573a641e2b55: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32987: connect: Connection reset by peer (error 104)
I20260620 08:58:39.232928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19789
I20260620 08:58:39.235015 20093 ts_tablet_manager.cc:1918] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:39.235117 20093 log.cc:1196] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945671001899-392-0/minicluster-data/ts-2/wal/wals/12abe3dbc7d446bca956573a641e2b55
I20260620 08:58:39.235404 20093 ts_tablet_manager.cc:1939] T 12abe3dbc7d446bca956573a641e2b55 P 2991671121b9427aae2acc3168558948: Deleting consensus metadata
I20260620 08:58:39.236599 19578 catalog_manager.cc:4841] TS 2991671121b9427aae2acc3168558948 (127.0.98.3:35815): tablet 12abe3dbc7d446bca956573a641e2b55 (table test-table [id=faa2f6a2e6df416980cccd4b47b8d8ec]) successfully deleted
I20260620 08:58:39.238874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19923
I20260620 08:58:39.243721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 19561
2026-06-20T08:58:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2147 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T08:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:39Z Disabled control of system clock
I20260620 08:58:39.272969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39173
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:58:39.384318 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:39.384562 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:39.384616 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:39.388605 20099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:39.388659 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:39.388674 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:39.388688 20099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:39.388702 20099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:39.393361 20099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:39.394312 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:39.395601 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:39.403657 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:39.403816 20099 server_base.cc:1034] running on GCE node
W20260620 08:58:39.403664 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.403698 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:39.404464 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:39.405511 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:39.406699 20099 hybrid_clock.cc:648] HybridClock initialized: now 1781945919406681 us; error 31 us; skew 500 ppm
I20260620 08:58:39.408473 20099 webserver.cc:460] Webserver started at http://127.0.98.62:41231/ using document root <none> and password file <none>
I20260620 08:58:39.408993 20099 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:39.409065 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:39.409256 20099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:39.411197 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "9ea5204bb61d463c8c5c90055f02189c"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "af5949950f49c767c5c53d4b5f5d66c6"
server_key_iv: "a465567ad8feced3f158eb269363abb5"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.411724 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ea5204bb61d463c8c5c90055f02189c"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "af5949950f49c767c5c53d4b5f5d66c6"
server_key_iv: "a465567ad8feced3f158eb269363abb5"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.415095 20099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:39.417827 20113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.419158 20099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:39.419253 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "9ea5204bb61d463c8c5c90055f02189c"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "af5949950f49c767c5c53d4b5f5d66c6"
server_key_iv: "a465567ad8feced3f158eb269363abb5"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.419332 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:39.428731 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:39.429283 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:39.429418 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:39.435701 20099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39173
I20260620 08:58:39.435710 20165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39173 every 8 connection(s)
I20260620 08:58:39.436743 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:39.437958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20099
I20260620 08:58:39.438074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:39.438340 392 external_mini_cluster.cc:1428] Setting key 857363bf2563ed4defef176175774cec
I20260620 08:58:39.440137 20166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:39.446739 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c: Bootstrap starting.
I20260620 08:58:39.448418 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.439539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.0.98.62:39173)
0620 08:58:39.440168 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.440187 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.440228 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:58:39.441045 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.441066 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.441519 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.441861 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:39.442280 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.443736 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.444751 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.445426 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.445605 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.446398 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:39.446405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:39.446420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:39.446895 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:39.446962 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:39.446988 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:39.447081 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.447233 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.447488 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260620 08:58:39.449326 20166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:39.450026 20166 log.cc:824] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:39.452229 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c: No bootstrap required, opened a new log
I20260620 08:58:39.455552 20166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea5204bb61d463c8c5c90055f02189c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39173 } }
I20260620 08:58:39.455814 20166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:39.455875 20166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea5204bb61d463c8c5c90055f02189c, State: Initialized, Role: FOLLOWER
I20260620 08:58:39.456274 20166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea5204bb61d463c8c5c90055f02189c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39173 } }
I20260620 08:58:39.456429 20166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:39.456494 20166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:39.456588 20166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:39.457430 20166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea5204bb61d463c8c5c90055f02189c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39173 } }
I20260620 08:58:39.457727 20166 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ea5204bb61d463c8c5c90055f02189c; no voters:
I20260620 08:58:39.458011 20166 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:39.458134 20171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:39.458338 20171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [term 1 LEADER]: Becoming Leader. State: Replica: 9ea5204bb61d463c8c5c90055f02189c, State: Running, Role: LEADER
I20260620 08:58:39.458788 20171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea5204bb61d463c8c5c90055f02189c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39173 } }
I20260620 08:58:39.459124 20166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:39.460270 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea5204bb61d463c8c5c90055f02189c. Latest consensus state: current_term: 1 leader_uuid: "9ea5204bb61d463c8c5c90055f02189c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea5204bb61d463c8c5c90055f02189c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39173 } } }
I20260620 08:58:39.460374 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:39.460574 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ea5204bb61d463c8c5c90055f02189c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea5204bb61d463c8c5c90055f02189c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39173 } } }
I20260620 08:58:39.460717 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:39.460765 20180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:39.463785 20180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:39.470688 20180 catalog_manager.cc:1347] Generated new cluster ID: adada1f46d84403e88f663374de58483
I20260620 08:58:39.470758 20180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:39.498029 20180 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:39.498926 20180 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:39.504925 20180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c: Generated new TSK 0
I20260620 08:58:39.505528 20180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:39.507965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39173
--builtin_ntp_servers=127.0.98.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:58:39.613933 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:39.614187 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:39.614238 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:39.618175 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:39.618225 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:39.618294 20190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:58:39.622970 20190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:39.623987 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:39.625276 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:39.634058 20198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.634083 20196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.634215 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:39.634274 20190 server_base.cc:1034] running on GCE node
I20260620 08:58:39.634656 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:39.635226 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:39.636371 20190 hybrid_clock.cc:648] HybridClock initialized: now 1781945919636357 us; error 25 us; skew 500 ppm
I20260620 08:58:39.638100 20190 webserver.cc:460] Webserver started at http://127.0.98.1:44069/ using document root <none> and password file <none>
I20260620 08:58:39.638622 20190 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:39.638680 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:39.638856 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:39.640826 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data/instance:
uuid: "80ea3e5652f04d26aa55d95f5056d2ce"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "8182a75f1570169596a6a21f720b653c"
server_key_iv: "8ed28e63ae758d432802d1c9900e4023"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.641335 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal/instance:
uuid: "80ea3e5652f04d26aa55d95f5056d2ce"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "8182a75f1570169596a6a21f720b653c"
server_key_iv: "8ed28e63ae758d432802d1c9900e4023"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.644860 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 08:58:39.647460 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.649065 20190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:39.649184 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal
uuid: "80ea3e5652f04d26aa55d95f5056d2ce"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "8182a75f1570169596a6a21f720b653c"
server_key_iv: "8ed28e63ae758d432802d1c9900e4023"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.649286 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:39.662189 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:39.662827 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:39.662998 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:39.663583 20190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:39.664518 20190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:39.664562 20190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.664623 20190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:39.664651 20190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.673514 20190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35217
I20260620 08:58:39.673532 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35217 every 8 connection(s)
I20260620 08:58:39.674403 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:58:39.683569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20190
I20260620 08:58:39.683724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:58:39.683916 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.676403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54705 (local address 127.0.98.62:39173)
0620 08:58:39.676594 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.676599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.677359 (+ 760us) server_negotiation.cc:407] Connection header received
0620 08:58:39.678642 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.678645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.678707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.678784 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:39.679701 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.680247 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.681796 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.682040 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.682125 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.683291 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:39.683293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:39.683296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:39.683309 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:39.683328 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:39.683335 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:39.683400 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.683658 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.683753 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 08:58:39.684031 392 external_mini_cluster.cc:1428] Setting key aba88d753f5a3cbfbc8c883558214f16
I20260620 08:58:39.684854 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.676490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39173 (local address 127.0.98.1:54705)
0620 08:58:39.677207 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.677247 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.678445 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.678985 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.678992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.679071 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:39.679530 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.679541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.680493 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.680498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.681644 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.681659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.682201 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.682205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.682356 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.683013 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:39.683015 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:39.683057 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:39.683119 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:39.683439 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:39.683449 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:39.683455 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.683588 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.683903 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":126,"threads_started":1}
I20260620 08:58:39.686061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39173
--builtin_ntp_servers=127.0.98.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:58:39.686095 20318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39173
I20260620 08:58:39.686339 20318 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:39.686884 20318 heartbeater.cc:511] Master 127.0.98.62:39173 requested a full tablet report, sending...
I20260620 08:58:39.688371 20130 ts_manager.cc:194] Registered new tserver with Master: 80ea3e5652f04d26aa55d95f5056d2ce (127.0.98.1:35217)
I20260620 08:58:39.689524 20130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54705
W20260620 08:58:39.793118 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:39.793352 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:39.793388 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:39.797363 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:39.797422 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:39.797478 20321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:58:39.802140 20321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:39.803104 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:39.804388 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:39.813230 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.813277 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.813230 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:39.813306 20321 server_base.cc:1034] running on GCE node
I20260620 08:58:39.813841 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:39.814445 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:39.815644 20321 hybrid_clock.cc:648] HybridClock initialized: now 1781945919815597 us; error 31 us; skew 500 ppm
I20260620 08:58:39.817394 20321 webserver.cc:460] Webserver started at http://127.0.98.2:39819/ using document root <none> and password file <none>
I20260620 08:58:39.817931 20321 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:39.817996 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:39.818172 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:39.820259 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data/instance:
uuid: "72db15f095f0447b960f6405f09be2d9"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "68fc4e46c459689661de71bfb4f5a6a7"
server_key_iv: "9956feb1cdb082739791e748eee99740"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.820825 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72db15f095f0447b960f6405f09be2d9"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "68fc4e46c459689661de71bfb4f5a6a7"
server_key_iv: "9956feb1cdb082739791e748eee99740"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.824436 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:39.826682 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.827824 20321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:39.827932 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal
uuid: "72db15f095f0447b960f6405f09be2d9"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "68fc4e46c459689661de71bfb4f5a6a7"
server_key_iv: "9956feb1cdb082739791e748eee99740"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.828019 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:39.838191 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:39.838794 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:39.838955 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:39.839493 20321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:39.840411 20321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:39.840458 20321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.840509 20321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:39.840538 20321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:39.848364 20321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46579
I20260620 08:58:39.848459 20448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46579 every 8 connection(s)
I20260620 08:58:39.849200 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:58:39.851944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20321
I20260620 08:58:39.852082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:58:39.852413 392 external_mini_cluster.cc:1428] Setting key 42d6646cee7342bc4bf45b959edf8c8d
I20260620 08:58:39.854331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39173
--builtin_ntp_servers=127.0.98.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:58:39.858902 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.850948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58891 (local address 127.0.98.62:39173)
0620 08:58:39.851081 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:39.851088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:39.852260 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 08:58:39.853841 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:39.853843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:39.853885 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:39.853948 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:39.855023 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.855623 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.856725 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.857045 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.857254 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.858312 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:39.858316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:39.858319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:39.858334 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:39.858360 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:39.858371 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:39.858453 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:39.858640 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 08:58:39.858710 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:58:39.859707 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:39.851290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39173 (local address 127.0.98.2:58891)
0620 08:58:39.852105 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:39.852157 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:39.853638 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:39.854120 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:39.854129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:39.854233 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:39.854859 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.854890 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.855750 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.855753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.856611 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:39.856619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:39.857279 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:39.857283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:39.857436 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:39.858054 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:39.858055 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:39.858094 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:39.858157 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:39.858482 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:39.858491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:39.858495 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:58:39.858609 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:58:39.858855 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":132,"threads_started":1}
I20260620 08:58:39.860785 20449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39173
I20260620 08:58:39.861018 20449 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:39.861500 20449 heartbeater.cc:511] Master 127.0.98.62:39173 requested a full tablet report, sending...
I20260620 08:58:39.862445 20130 ts_manager.cc:194] Registered new tserver with Master: 72db15f095f0447b960f6405f09be2d9 (127.0.98.2:46579)
I20260620 08:58:39.862995 20130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58891
W20260620 08:58:39.960950 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:39.961196 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:39.961249 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:39.965675 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:39.965903 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:39.966046 20452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:58:39.972039 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:39.973129 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:39.974475 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:39.984138 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.984136 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:39.984138 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:39.984382 20452 server_base.cc:1034] running on GCE node
I20260620 08:58:39.984860 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:39.985442 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:39.986615 20452 hybrid_clock.cc:648] HybridClock initialized: now 1781945919986595 us; error 32 us; skew 500 ppm
I20260620 08:58:39.988441 20452 webserver.cc:460] Webserver started at http://127.0.98.3:43089/ using document root <none> and password file <none>
I20260620 08:58:39.988977 20452 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:39.989048 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:39.989223 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:39.991254 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d1f0da8fc3544e297848f7605934028"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "83a3ac1d6f21dfbcfdacff9e101ec504"
server_key_iv: "6a22b3159320637f904a8a4e3ff3e8ab"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.991815 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d1f0da8fc3544e297848f7605934028"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "83a3ac1d6f21dfbcfdacff9e101ec504"
server_key_iv: "6a22b3159320637f904a8a4e3ff3e8ab"
server_key_version: "encryptionkey@0"
I20260620 08:58:39.996189 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 08:58:39.999507 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:40.001070 20452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:40.001196 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal
uuid: "9d1f0da8fc3544e297848f7605934028"
format_stamp: "Formatted at 2026-06-20 08:58:39 on dist-test-slave-fb1v"
server_key: "83a3ac1d6f21dfbcfdacff9e101ec504"
server_key_iv: "6a22b3159320637f904a8a4e3ff3e8ab"
server_key_version: "encryptionkey@0"
I20260620 08:58:40.001309 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:40.016095 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:40.016794 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:40.016963 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:40.017540 20452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:58:40.018813 20452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:58:40.019028 20452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:40.019131 20452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:58:40.019160 20452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:40.029678 20452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45341
I20260620 08:58:40.029877 20579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45341 every 8 connection(s)
I20260620 08:58:40.030690 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:58:40.039821 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.032602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43225 (local address 127.0.98.62:39173)
0620 08:58:40.032756 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.032759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.033692 (+ 933us) server_negotiation.cc:407] Connection header received
0620 08:58:40.035021 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.035025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.035104 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.035176 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:40.036264 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.036784 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.037905 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.038123 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.038210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.039261 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:40.039264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:40.039266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:40.039280 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:40.039301 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:40.039311 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:40.039382 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.039533 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.039653 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 08:58:40.040691 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.032870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39173 (local address 127.0.98.3:43225)
0620 08:58:40.033506 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.033563 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:40.034816 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:40.035346 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:40.035353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:40.035435 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:58:40.036120 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.036134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.036898 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.036901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.037791 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.037800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.038260 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.038264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.038413 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.038988 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:58:40.038989 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:58:40.039029 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:58:40.039093 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:58:40.039392 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:58:40.039401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:58:40.039408 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:58:40.039525 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 08:58:40.039844 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260620 08:58:40.040747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20452
I20260620 08:58:40.040860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:58:40.041144 392 external_mini_cluster.cc:1428] Setting key a9898637450bf596d786d5b43a34ef2e
I20260620 08:58:40.041831 20580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39173
I20260620 08:58:40.042080 20580 heartbeater.cc:464] Registering TS with master...
I20260620 08:58:40.042632 20580 heartbeater.cc:511] Master 127.0.98.62:39173 requested a full tablet report, sending...
I20260620 08:58:40.043725 20130 ts_manager.cc:194] Registered new tserver with Master: 9d1f0da8fc3544e297848f7605934028 (127.0.98.3:45341)
I20260620 08:58:40.044272 20130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43225
I20260620 08:58:40.055893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:58:40.061578 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.057931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.0.98.62:39173)
0620 08:58:40.058063 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.058066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.058110 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:58:40.058222 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.058225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.058269 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.058341 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:40.058642 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.059181 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.060054 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.060272 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.060428 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.061179 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:40.061182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:40.061184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:40.061196 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:40.061214 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:40.061221 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:40.061281 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.061405 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.061456 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:58:40.064618 20130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:58:40.067710 20130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:58:40.081653 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.076318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.98.3:45341)
0620 08:58:40.077027 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.077038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.077062 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:58:40.077140 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.077146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.077343 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.077498 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:40.078002 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.078817 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.080208 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.081013 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.081109 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.081309 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.081393 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.081486 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":143,"threads_started":1}
I20260620 08:58:40.081648 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.076200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45341 (local address 127.0.0.1:32808)
0620 08:58:40.076544 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.076594 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:40.076718 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:40.077705 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:40.077711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:40.077734 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:40.077876 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.077883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.078921 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.078924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.080064 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.080084 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.081110 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.081113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.081224 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.081260 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:58:40.081362 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:58:40.081484 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260620 08:58:40.082260 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.076276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35217 (local address 127.0.0.1:38720)
0620 08:58:40.076872 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.076886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:40.076972 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:40.077965 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:40.077968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:40.077986 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:40.078080 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.078085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.079770 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.079774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.080863 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.080874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.081910 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.081914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.081997 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.082020 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:58:40.082071 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:58:40.082131 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":85,"threads_started":1}
I20260620 08:58:40.082912 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.076988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.98.1:35217)
0620 08:58:40.077530 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.077542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.077565 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:58:40.077640 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.077645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.077823 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.077967 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:40.078337 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.079666 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.080990 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.081781 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.082433 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.082617 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.082668 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.082759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":113,"threads_started":1}
I20260620 08:58:40.083635 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.077507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.0.98.2:46579)
0620 08:58:40.078391 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.078420 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.078454 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:58:40.078555 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.078561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.078957 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.079124 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:40.079322 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.080355 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.081544 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.082858 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.082966 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.083130 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.083346 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.083441 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":130,"threads_started":1}
I20260620 08:58:40.083916 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.077068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46579 (local address 127.0.0.1:48086)
0620 08:58:40.077881 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.077893 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:40.077965 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:40.079110 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:40.079112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:40.079131 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:40.079205 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.079211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.080468 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.080471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.081410 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.081420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.083128 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.083131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.083215 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.083238 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:58:40.083313 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:58:40.083641 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":99,"threads_started":1}
I20260620 08:58:40.084750 20514 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 (DEFAULT_TABLE table=test-table [id=9b5e13e8176640e4a1fb7ac2ae3a3189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:40.085912 20514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:40.086385 20383 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 (DEFAULT_TABLE table=test-table [id=9b5e13e8176640e4a1fb7ac2ae3a3189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:40.086517 20252 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 (DEFAULT_TABLE table=test-table [id=9b5e13e8176640e4a1fb7ac2ae3a3189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:58:40.087316 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:40.087437 20252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:40.093657 20598 tablet_bootstrap.cc:492] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Bootstrap starting.
I20260620 08:58:40.094295 20599 tablet_bootstrap.cc:492] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: Bootstrap starting.
I20260620 08:58:40.095556 20598 tablet_bootstrap.cc:654] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:40.095961 20599 tablet_bootstrap.cc:654] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:40.095991 20597 tablet_bootstrap.cc:492] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Bootstrap starting.
I20260620 08:58:40.096278 20598 log.cc:824] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:40.096741 20599 log.cc:824] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:40.097898 20597 tablet_bootstrap.cc:654] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:40.098023 20598 tablet_bootstrap.cc:492] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: No bootstrap required, opened a new log
I20260620 08:58:40.098194 20598 ts_tablet_manager.cc:1397] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:58:40.098673 20597 log.cc:824] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:40.099090 20599 tablet_bootstrap.cc:492] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: No bootstrap required, opened a new log
I20260620 08:58:40.099269 20599 ts_tablet_manager.cc:1397] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:58:40.100597 20597 tablet_bootstrap.cc:492] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: No bootstrap required, opened a new log
I20260620 08:58:40.100772 20597 ts_tablet_manager.cc:1397] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:58:40.101890 20598 raft_consensus.cc:348] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.102101 20598 raft_consensus.cc:374] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:40.102151 20598 raft_consensus.cc:729] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1f0da8fc3544e297848f7605934028, State: Initialized, Role: FOLLOWER
I20260620 08:58:40.102550 20598 consensus_queue.cc:260] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.103168 20580 heartbeater.cc:503] Master 127.0.98.62:39173 was elected leader, sending a full tablet report...
I20260620 08:58:40.103418 20598 ts_tablet_manager.cc:1428] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:58:40.102985 20599 raft_consensus.cc:348] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.103854 20599 raft_consensus.cc:374] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:40.103914 20599 raft_consensus.cc:729] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ea3e5652f04d26aa55d95f5056d2ce, State: Initialized, Role: FOLLOWER
I20260620 08:58:40.104326 20599 consensus_queue.cc:260] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.104786 20597 raft_consensus.cc:348] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.104971 20318 heartbeater.cc:503] Master 127.0.98.62:39173 was elected leader, sending a full tablet report...
I20260620 08:58:40.105115 20597 raft_consensus.cc:374] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:40.105170 20597 raft_consensus.cc:729] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72db15f095f0447b960f6405f09be2d9, State: Initialized, Role: FOLLOWER
I20260620 08:58:40.105754 20599 ts_tablet_manager.cc:1428] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 08:58:40.105602 20597 consensus_queue.cc:260] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.106586 20597 ts_tablet_manager.cc:1428] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 08:58:40.106772 20449 heartbeater.cc:503] Master 127.0.98.62:39173 was elected leader, sending a full tablet report...
W20260620 08:58:40.175504 20319 tablet.cc:2369] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:58:40.281720 20581 tablet.cc:2369] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:40.340627 20603 raft_consensus.cc:482] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:58:40.340837 20603 raft_consensus.cc:504] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.342010 20603 leader_election.cc:288] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80ea3e5652f04d26aa55d95f5056d2ce (127.0.98.1:35217), 72db15f095f0447b960f6405f09be2d9 (127.0.98.2:46579)
I20260620 08:58:40.346835 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.342396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39945 (local address 127.0.98.1:35217)
0620 08:58:40.342511 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.342516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.342604 (+ 88us) server_negotiation.cc:407] Connection header received
0620 08:58:40.342747 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.342750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.342800 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.342865 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:40.343135 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.343924 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.345497 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.346144 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.346232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.346285 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.346635 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.346714 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:58:40.346835 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.342376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39105 (local address 127.0.98.2:46579)
0620 08:58:40.342510 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.342515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.342903 (+ 388us) server_negotiation.cc:407] Connection header received
0620 08:58:40.343022 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.343025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.343068 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.343136 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:58:40.343351 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.344065 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.345497 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.346112 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.346223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.346285 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.346640 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.346714 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:58:40.346836 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.342279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35217 (local address 127.0.98.3:39945)
0620 08:58:40.342522 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.342537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:40.342637 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:40.342897 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:40.342900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:40.342918 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:40.343039 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.343046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.344034 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.344037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.345357 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.345368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.346281 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.346285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.346458 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.346512 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 08:58:40.346589 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:58:40.346685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":103}
I20260620 08:58:40.346848 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.342279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46579 (local address 127.0.98.3:39105)
0620 08:58:40.342804 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.342814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:58:40.342934 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:58:40.343156 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:58:40.343158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:58:40.343171 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:58:40.343255 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.343259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.344155 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.344158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.345357 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:58:40.345368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.346281 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.346285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:58:40.346437 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.346503 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 08:58:40.346587 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 08:58:40.346685 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":237,"spinlock_wait_cycles":2944,"thread_start_us":126,"threads_started":1}
I20260620 08:58:40.347651 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf5d86797cb4ade8c12fc1915ea4fa0" candidate_uuid: "9d1f0da8fc3544e297848f7605934028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72db15f095f0447b960f6405f09be2d9" is_pre_election: true
I20260620 08:58:40.347934 20403 raft_consensus.cc:2454] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1f0da8fc3544e297848f7605934028 in term 0.
I20260620 08:58:40.348155 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf5d86797cb4ade8c12fc1915ea4fa0" candidate_uuid: "9d1f0da8fc3544e297848f7605934028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" is_pre_election: true
I20260620 08:58:40.348536 20470 leader_election.cc:302] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72db15f095f0447b960f6405f09be2d9, 9d1f0da8fc3544e297848f7605934028; no voters:
I20260620 08:58:40.348568 20272 raft_consensus.cc:2454] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1f0da8fc3544e297848f7605934028 in term 0.
I20260620 08:58:40.348909 20603 raft_consensus.cc:2781] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:58:40.349014 20603 raft_consensus.cc:482] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:58:40.349068 20603 raft_consensus.cc:3037] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:40.350085 20603 raft_consensus.cc:504] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
W20260620 08:58:40.350337 20450 tablet.cc:2369] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:58:40.350432 20603 leader_election.cc:288] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [CANDIDATE]: Term 1 election: Requested vote from peers 80ea3e5652f04d26aa55d95f5056d2ce (127.0.98.1:35217), 72db15f095f0447b960f6405f09be2d9 (127.0.98.2:46579)
I20260620 08:58:40.350867 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf5d86797cb4ade8c12fc1915ea4fa0" candidate_uuid: "9d1f0da8fc3544e297848f7605934028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ea3e5652f04d26aa55d95f5056d2ce"
I20260620 08:58:40.351050 20272 raft_consensus.cc:3037] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:40.351679 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf5d86797cb4ade8c12fc1915ea4fa0" candidate_uuid: "9d1f0da8fc3544e297848f7605934028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72db15f095f0447b960f6405f09be2d9"
I20260620 08:58:40.351806 20403 raft_consensus.cc:3037] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:40.352373 20272 raft_consensus.cc:2454] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1f0da8fc3544e297848f7605934028 in term 1.
I20260620 08:58:40.352728 20468 leader_election.cc:302] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ea3e5652f04d26aa55d95f5056d2ce, 9d1f0da8fc3544e297848f7605934028; no voters:
I20260620 08:58:40.352907 20603 raft_consensus.cc:2781] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:40.353049 20403 raft_consensus.cc:2454] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1f0da8fc3544e297848f7605934028 in term 1.
I20260620 08:58:40.353109 20603 raft_consensus.cc:686] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1f0da8fc3544e297848f7605934028, State: Running, Role: LEADER
I20260620 08:58:40.353467 20603 consensus_queue.cc:237] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } }
I20260620 08:58:40.356456 20128 catalog_manager.cc:5495] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d1f0da8fc3544e297848f7605934028 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9d1f0da8fc3544e297848f7605934028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1f0da8fc3544e297848f7605934028" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 } health_report { overall_health: UNKNOWN } } }
I20260620 08:58:40.375104 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.371908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.0.98.3:45341)
0620 08:58:40.372025 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:40.372030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:40.372042 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:58:40.372112 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:40.372114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:40.372164 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:40.372251 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:58:40.372460 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.372963 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.373772 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:40.373977 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:40.374061 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:40.374331 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:58:40.374734 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:58:40.374833 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:40.374907 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 08:58:40.374959 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:58:40.380609 20272 raft_consensus.cc:1261] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 1 FOLLOWER]: Refusing update from remote peer 9d1f0da8fc3544e297848f7605934028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:40.380625 20403 raft_consensus.cc:1261] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d1f0da8fc3544e297848f7605934028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:58:40.381245 20603 consensus_queue.cc:1035] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ea3e5652f04d26aa55d95f5056d2ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:40.381421 20607 consensus_queue.cc:1035] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72db15f095f0447b960f6405f09be2d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:58:40.387513 20610 mvcc.cc:205] Tried to move back new op lower bound from 7298850489872265216 to 7298850489770315776. Current Snapshot: MvccSnapshot[applied={T|T < 7298850489872265216}]
I20260620 08:58:40.395650 20128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:40.395833 20128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:58:40.398252 20128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ea5204bb61d463c8c5c90055f02189c: Sending DeleteTablet for 3 replicas of tablet 9bf5d86797cb4ade8c12fc1915ea4fa0
I20260620 08:58:40.398967 20514 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:40 UTC) from {username='slave'} at 127.0.0.1:32808
I20260620 08:58:40.399006 20252 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:40 UTC) from {username='slave'} at 127.0.0.1:38720
I20260620 08:58:40.399109 20383 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:58:40 UTC) from {username='slave'} at 127.0.0.1:48086
I20260620 08:58:40.399490 20621 tablet_replica.cc:330] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce: stopping tablet replica
I20260620 08:58:40.399664 20622 tablet_replica.cc:330] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: stopping tablet replica
I20260620 08:58:40.399845 20621 raft_consensus.cc:2229] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:40.400053 20622 raft_consensus.cc:2229] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:58:40.400198 20621 raft_consensus.cc:2258] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 80ea3e5652f04d26aa55d95f5056d2ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:40.400235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20190
I20260620 08:58:40.400453 20622 raft_consensus.cc:2258] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:40.401861 20622 ts_tablet_manager.cc:1905] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:40.404580 20622 ts_tablet_manager.cc:1918] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:58:40.404688 20622 log.cc:1196] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Deleting WAL directory at /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945671001899-392-0/minicluster-data/ts-1/wal/wals/9bf5d86797cb4ade8c12fc1915ea4fa0
I20260620 08:58:40.404956 20622 ts_tablet_manager.cc:1939] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 72db15f095f0447b960f6405f09be2d9: Deleting consensus metadata
I20260620 08:58:40.405503 20620 tablet_replica.cc:330] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: stopping tablet replica
I20260620 08:58:40.405829 20620 raft_consensus.cc:2229] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:58:40.406314 20620 raft_consensus.cc:2258] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:58:40.406706 20117 catalog_manager.cc:4841] TS 72db15f095f0447b960f6405f09be2d9 (127.0.98.2:46579): tablet 9bf5d86797cb4ade8c12fc1915ea4fa0 (table test-table [id=9b5e13e8176640e4a1fb7ac2ae3a3189]) successfully deleted
W20260620 08:58:40.408090 20115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35217 (error 108)
I20260620 08:58:40.408188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20321
I20260620 08:58:40.412405 20620 ts_tablet_manager.cc:1905] T 9bf5d86797cb4ade8c12fc1915ea4fa0 P 9d1f0da8fc3544e297848f7605934028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:58:40.412447 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:40.412158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35217 (local address 127.0.0.1:38724)
0620 08:58:40.412290 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 08:58:40.412360 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 08:58:40.412613 20115 catalog_manager.cc:4584] TS 80ea3e5652f04d26aa55d95f5056d2ce (127.0.98.1:35217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bf5d86797cb4ade8c12fc1915ea4fa0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35217: connect: Connection refused (error 111)
I20260620 08:58:40.414933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20452
I20260620 08:58:40.419927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20099
2026-06-20T08:58:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1174 ms)
[----------] 6 tests from EncryptionPolicyTest (9906 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:40 dist-test-slave-fb1v krb5kdc[20626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:40 dist-test-slave-fb1v krb5kdc[20626](info): set up 2 sockets
Jun 20 08:58:40 dist-test-slave-fb1v krb5kdc[20626](info): commencing operation
krb5kdc: starting...
W20260620 08:58:41.375420 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:41 dist-test-slave-fb1v krb5kdc[20626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:41.498451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:41.607396 20639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:41.607690 20639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:41.607733 20639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:41.611688 20639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:41.611747 20639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:41.611765 20639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:41.611781 20639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:41.611795 20639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:41.617780 20639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:41.618808 20639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:41.619788 20639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:41.627885 20644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:41.627897 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:41.627899 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:41.628127 20639 server_base.cc:1034] running on GCE node
I20260620 08:58:41.628602 20639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:41.629626 20639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:41.630795 20639 hybrid_clock.cc:648] HybridClock initialized: now 1781945921630764 us; error 47 us; skew 500 ppm
Jun 20 08:58:41 dist-test-slave-fb1v krb5kdc[20626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:41.633602 20639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:41.634486 20639 webserver.cc:460] Webserver started at http://127.0.0.1:35829/ using document root <none> and password file <none>
I20260620 08:58:41.635011 20639 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:41.635068 20639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:41.635246 20639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:41.637234 20639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "4a986e3b41a04493801cb9a185060b29"
format_stamp: "Formatted at 2026-06-20 08:58:41 on dist-test-slave-fb1v"
server_key: "ffd9ed6ca25513a36575adb0c272acdb"
server_key_iv: "e7f63a97248dcbed070b0ee193bc8982"
server_key_version: "encryptionkey@0"
I20260620 08:58:41.637753 20639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a986e3b41a04493801cb9a185060b29"
format_stamp: "Formatted at 2026-06-20 08:58:41 on dist-test-slave-fb1v"
server_key: "ffd9ed6ca25513a36575adb0c272acdb"
server_key_iv: "e7f63a97248dcbed070b0ee193bc8982"
server_key_version: "encryptionkey@0"
I20260620 08:58:41.641453 20639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 08:58:41.643839 20654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:41.644994 20639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:58:41.645095 20639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "4a986e3b41a04493801cb9a185060b29"
format_stamp: "Formatted at 2026-06-20 08:58:41 on dist-test-slave-fb1v"
server_key: "ffd9ed6ca25513a36575adb0c272acdb"
server_key_iv: "e7f63a97248dcbed070b0ee193bc8982"
server_key_version: "encryptionkey@0"
I20260620 08:58:41.645191 20639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:41.670207 20639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:41.673323 20639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:41.673511 20639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:41.680516 20639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34601
I20260620 08:58:41.680531 20706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34601 every 8 connection(s)
I20260620 08:58:41.681444 20639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:41.684110 20707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:41.684854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20639
I20260620 08:58:41.684962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:41.685246 392 external_mini_cluster.cc:1428] Setting key d5f3c746887f39894f5f879ae85886f1
I20260620 08:58:41.690407 20707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29: Bootstrap starting.
I20260620 08:58:41.692939 20707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:41.693809 20707 log.cc:824] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:58:41 dist-test-slave-fb1v krb5kdc[20626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:58:41.696157 20707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29: No bootstrap required, opened a new log
I20260620 08:58:41.700395 20707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986e3b41a04493801cb9a185060b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260620 08:58:41.700636 20707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:41.700685 20707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a986e3b41a04493801cb9a185060b29, State: Initialized, Role: FOLLOWER
I20260620 08:58:41.701144 20707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986e3b41a04493801cb9a185060b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260620 08:58:41.701267 20707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:41.701329 20707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:41.701401 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:41.686952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39711 (local address 127.0.0.1:34601)
0620 08:58:41.687433 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:41.687449 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:41.687487 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:58:41.688160 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:41.688177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:41.688528 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:41.688821 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:41.689793 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:41.690879 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:41.692048 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:41.692528 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:41.692671 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:41.695525 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:41.695541 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:41.695554 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:41.695583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:41.697784 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:41.698377 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:41.698381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:41.698386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:41.698499 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:41.698906 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:41.698911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:41.698913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:41.699561 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:41.699904 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:41.700274 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 08:58:41.700533 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260620 08:58:41.701419 20707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:41.702473 20707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986e3b41a04493801cb9a185060b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260620 08:58:41.702824 20707 leader_election.cc:302] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a986e3b41a04493801cb9a185060b29; no voters:
I20260620 08:58:41.703155 20707 leader_election.cc:288] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:41.703258 20712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:41.703464 20712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [term 1 LEADER]: Becoming Leader. State: Replica: 4a986e3b41a04493801cb9a185060b29, State: Running, Role: LEADER
I20260620 08:58:41.703815 20712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986e3b41a04493801cb9a185060b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260620 08:58:41.704447 20707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:41.704645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:41.705353 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a986e3b41a04493801cb9a185060b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986e3b41a04493801cb9a185060b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } } }
I20260620 08:58:41.705480 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:41.705875 20721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:41.706876 20714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a986e3b41a04493801cb9a185060b29. Latest consensus state: current_term: 1 leader_uuid: "4a986e3b41a04493801cb9a185060b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a986e3b41a04493801cb9a185060b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } } }
I20260620 08:58:41.707036 20714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:41.709017 20721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:41.715176 20721 catalog_manager.cc:1347] Generated new cluster ID: ec8ce3d4f0cb4c7889984901036116ef
I20260620 08:58:41.715240 20721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:41.715927 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:41.706854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33395 (local address 127.0.0.1:34601)
0620 08:58:41.706956 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:41.706960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:41.707449 (+ 489us) server_negotiation.cc:407] Connection header received
0620 08:58:41.707638 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:41.707641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:41.707690 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:41.707764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:41.708563 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:41.709290 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:41.710499 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:41.710837 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:41.710932 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:41.712280 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:41.712284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:41.712287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:41.712312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:41.713978 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:41.714544 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:41.714547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:41.714549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:41.714630 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:41.714984 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:41.714987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:41.714989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:41.715207 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:41.715399 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:41.715701 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 08:58:41.715786 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:58:41.748241 20721 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:41.749233 20721 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:41.758397 20721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a986e3b41a04493801cb9a185060b29: Generated new TSK 0
I20260620 08:58:41.759181 20721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:41.774753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20639
W20260620 08:58:41.779908 20624 connection.cc:535] client connection to 127.0.0.1:34601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34601): Connection reset by peer (error 104)
2026-06-20T08:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:41 dist-test-slave-fb1v krb5kdc[20738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:41 dist-test-slave-fb1v krb5kdc[20738](info): set up 2 sockets
Jun 20 08:58:41 dist-test-slave-fb1v krb5kdc[20738](info): commencing operation
krb5kdc: starting...
W20260620 08:58:42.732008 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:42 dist-test-slave-fb1v krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:42.846720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:42.954107 20751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:42.954365 20751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:42.954407 20751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:42.958658 20751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:42.958725 20751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:42.958743 20751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:42.958760 20751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:42.958786 20751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:42.964184 20751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:42.965197 20751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:42.966364 20751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:42.975297 20759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:42.975317 20756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:42.975297 20757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:42.975384 20751 server_base.cc:1034] running on GCE node
I20260620 08:58:42.976179 20751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:42.977254 20751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:42.978415 20751 hybrid_clock.cc:648] HybridClock initialized: now 1781945922978400 us; error 26 us; skew 500 ppm
Jun 20 08:58:42 dist-test-slave-fb1v krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:42.985989 20751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:42.987010 20751 webserver.cc:460] Webserver started at http://127.0.0.1:40699/ using document root <none> and password file <none>
I20260620 08:58:42.987568 20751 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:42.987661 20751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:42.987895 20751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:42.990016 20751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "bc6d7c847b23425d8dc72ec97d4e3d00"
format_stamp: "Formatted at 2026-06-20 08:58:42 on dist-test-slave-fb1v"
server_key: "d9a915cfafd24b4e0ac3d5a11d40c24a"
server_key_iv: "9c14d7a1015ec644e4b805b4ac86ceae"
server_key_version: "encryptionkey@0"
I20260620 08:58:42.990571 20751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc6d7c847b23425d8dc72ec97d4e3d00"
format_stamp: "Formatted at 2026-06-20 08:58:42 on dist-test-slave-fb1v"
server_key: "d9a915cfafd24b4e0ac3d5a11d40c24a"
server_key_iv: "9c14d7a1015ec644e4b805b4ac86ceae"
server_key_version: "encryptionkey@0"
I20260620 08:58:42.994596 20751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 08:58:42.997498 20766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:42.998862 20751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:58:42.998993 20751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "bc6d7c847b23425d8dc72ec97d4e3d00"
format_stamp: "Formatted at 2026-06-20 08:58:42 on dist-test-slave-fb1v"
server_key: "d9a915cfafd24b4e0ac3d5a11d40c24a"
server_key_iv: "9c14d7a1015ec644e4b805b4ac86ceae"
server_key_version: "encryptionkey@0"
I20260620 08:58:42.999106 20751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:43.018263 20751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:43.021570 20751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:43.021770 20751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:43.028781 20751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39305
I20260620 08:58:43.028834 20818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39305 every 8 connection(s)
I20260620 08:58:43.029698 20751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:43.032281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20751
I20260620 08:58:43.032411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:43.032434 20819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:43.032676 392 external_mini_cluster.cc:1428] Setting key f3833fe585f8616420e9ff8b376ae860
I20260620 08:58:43.038874 20819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00: Bootstrap starting.
I20260620 08:58:43.040953 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:43.033852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41459 (local address 127.0.0.1:39305)
0620 08:58:43.034234 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:43.034249 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:43.034281 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:58:43.034872 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:43.034887 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:43.035223 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:43.035478 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:43.035831 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:43.036778 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:43.037941 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:43.038397 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:43.038507 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:43.039366 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:43.039371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:43.039382 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:43.039778 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:43.039830 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:43.039853 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:43.039950 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:43.040151 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 08:58:43.040281 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260620 08:58:43.041132 20819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:43.041967 20819 log.cc:824] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:43.043838 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:43.043879 20819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00: No bootstrap required, opened a new log
I20260620 08:58:43.046998 20819 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260620 08:58:43.047252 20819 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:43.047313 20819 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6d7c847b23425d8dc72ec97d4e3d00, State: Initialized, Role: FOLLOWER
I20260620 08:58:43.047771 20819 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260620 08:58:43.047917 20819 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:43.047982 20819 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:43.048080 20819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:43.049029 20819 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260620 08:58:43.049348 20819 leader_election.cc:302] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc6d7c847b23425d8dc72ec97d4e3d00; no voters:
I20260620 08:58:43.049667 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:43.045926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59583 (local address 127.0.0.1:39305)
0620 08:58:43.046027 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:43.046030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:43.046040 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:58:43.046070 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:43.046072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:43.046103 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:43.046164 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:43.046459 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:43.046979 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:43.048068 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:43.048352 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:43.048432 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:43.049234 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:43.049238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:43.049241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:43.049252 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:43.049275 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:43.049285 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:43.049363 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:43.049503 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 08:58:43.049542 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:58:43.049680 20819 leader_election.cc:288] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:43.049741 20829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:43.049964 20829 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [term 1 LEADER]: Becoming Leader. State: Replica: bc6d7c847b23425d8dc72ec97d4e3d00, State: Running, Role: LEADER
I20260620 08:58:43.050290 20829 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260620 08:58:43.050920 20819 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:43.052057 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc6d7c847b23425d8dc72ec97d4e3d00. Latest consensus state: current_term: 1 leader_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } } }
I20260620 08:58:43.052073 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc6d7c847b23425d8dc72ec97d4e3d00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } } }
I20260620 08:58:43.052253 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:43.052263 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00 [sys.catalog]: This master's current role is: LEADER
W20260620 08:58:43.055377 20844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:58:43.055438 20844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:58:43.055516 20839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:43.056205 20839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:43.060969 20839 catalog_manager.cc:1347] Generated new cluster ID: 10abdc1d681b43228363c838feceb4f0
I20260620 08:58:43.061046 20839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:43.073063 20839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:43.074271 20839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:43.079034 20839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc6d7c847b23425d8dc72ec97d4e3d00: Generated new TSK 0
I20260620 08:58:43.079808 20839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:43.088445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20751
W20260620 08:58:43.092919 20736 connection.cc:535] client connection to 127.0.0.1:39305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39305): Connection reset by peer (error 104)
2026-06-20T08:58:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:43 dist-test-slave-fb1v krb5kdc[20850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:43 dist-test-slave-fb1v krb5kdc[20850](info): set up 2 sockets
Jun 20 08:58:43 dist-test-slave-fb1v krb5kdc[20850](info): commencing operation
krb5kdc: starting...
W20260620 08:58:44.043566 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:44 dist-test-slave-fb1v krb5kdc[20850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:44.158569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:44.264612 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:44.264892 20863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:44.264933 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:44.268862 20863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:44.268916 20863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:44.268934 20863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:44.268949 20863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:44.268962 20863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:44.274111 20863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:44.275068 20863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:44.276017 20863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:44.283943 20871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:44.284060 20869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:44.283943 20868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:44.284183 20863 server_base.cc:1034] running on GCE node
I20260620 08:58:44.284839 20863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:44.285943 20863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:44.287127 20863 hybrid_clock.cc:648] HybridClock initialized: now 1781945924287076 us; error 60 us; skew 500 ppm
Jun 20 08:58:44 dist-test-slave-fb1v krb5kdc[20850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:44.290004 20863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:44.290936 20863 webserver.cc:460] Webserver started at http://127.0.0.1:37777/ using document root <none> and password file <none>
I20260620 08:58:44.291505 20863 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:44.291587 20863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:44.291842 20863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:44.293848 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "65b422a50d404bd2be3d06d94b31361e"
format_stamp: "Formatted at 2026-06-20 08:58:44 on dist-test-slave-fb1v"
server_key: "40e9dcdc74b3b017dd1bfa61494216a0"
server_key_iv: "f1955e72dd745792993e7c5efb172c03"
server_key_version: "encryptionkey@0"
I20260620 08:58:44.294348 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "65b422a50d404bd2be3d06d94b31361e"
format_stamp: "Formatted at 2026-06-20 08:58:44 on dist-test-slave-fb1v"
server_key: "40e9dcdc74b3b017dd1bfa61494216a0"
server_key_iv: "f1955e72dd745792993e7c5efb172c03"
server_key_version: "encryptionkey@0"
I20260620 08:58:44.297824 20863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:44.300181 20878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:44.301494 20863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:44.301617 20863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "65b422a50d404bd2be3d06d94b31361e"
format_stamp: "Formatted at 2026-06-20 08:58:44 on dist-test-slave-fb1v"
server_key: "40e9dcdc74b3b017dd1bfa61494216a0"
server_key_iv: "f1955e72dd745792993e7c5efb172c03"
server_key_version: "encryptionkey@0"
I20260620 08:58:44.301723 20863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:44.306160 20863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:44.309135 20863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:44.309298 20863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:44.316011 20863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46445
I20260620 08:58:44.316025 20930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46445 every 8 connection(s)
I20260620 08:58:44.316936 20863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:44.319674 20931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:44.324446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20863
I20260620 08:58:44.324585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:44.324877 392 external_mini_cluster.cc:1428] Setting key 6ac3f6f65e999a3df731d04b63683c8a
I20260620 08:58:44.325261 20931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e: Bootstrap starting.
I20260620 08:58:44.327845 20931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:44.328756 20931 log.cc:824] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:44.331027 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:44.326323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33903 (local address 127.0.0.1:46445)
0620 08:58:44.326746 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:44.326764 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:44.326804 (+ 40us) server_negotiation.cc:407] Connection header received
0620 08:58:44.327654 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:44.327677 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:44.328049 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:44.328413 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:44.329510 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:44.329515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:44.329527 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:44.329539 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:44.329601 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:44.329622 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:44.329723 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:44.329941 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 08:58:44.330131 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260620 08:58:44.331369 20931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e: No bootstrap required, opened a new log
I20260620 08:58:44.334323 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:44.335723 20931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b422a50d404bd2be3d06d94b31361e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260620 08:58:44.335943 20931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:44.335999 20931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b422a50d404bd2be3d06d94b31361e, State: Initialized, Role: FOLLOWER
I20260620 08:58:44.336498 20931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b422a50d404bd2be3d06d94b31361e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260620 08:58:44.336640 20931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:44.336738 20931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:44.336839 20931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:44.337841 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:44.336295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51837 (local address 127.0.0.1:46445)
0620 08:58:44.336405 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:44.336409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:44.336605 (+ 196us) server_negotiation.cc:407] Connection header received
0620 08:58:44.336727 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:44.336730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:44.336772 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:44.336856 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:44.337505 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:44.337508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:44.337510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:44.337522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:44.337551 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:44.337563 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:44.337648 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:44.337696 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 08:58:44.337732 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:58:44.337850 20931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b422a50d404bd2be3d06d94b31361e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260620 08:58:44.338210 20931 leader_election.cc:302] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b422a50d404bd2be3d06d94b31361e; no voters:
I20260620 08:58:44.338495 20931 leader_election.cc:288] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:44.338562 20941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:44.338724 20941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [term 1 LEADER]: Becoming Leader. State: Replica: 65b422a50d404bd2be3d06d94b31361e, State: Running, Role: LEADER
I20260620 08:58:44.339010 20941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b422a50d404bd2be3d06d94b31361e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260620 08:58:44.339753 20931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:44.340152 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b422a50d404bd2be3d06d94b31361e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b422a50d404bd2be3d06d94b31361e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } } }
I20260620 08:58:44.340263 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:44.340481 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b422a50d404bd2be3d06d94b31361e. Latest consensus state: current_term: 1 leader_uuid: "65b422a50d404bd2be3d06d94b31361e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b422a50d404bd2be3d06d94b31361e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } } }
I20260620 08:58:44.340596 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:44.340940 20945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:44.344023 20945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:44.350045 20945 catalog_manager.cc:1347] Generated new cluster ID: 407fd6e0a4ed42d3810bf588b676d5c7
I20260620 08:58:44.350126 20945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:44.359129 20945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:44.359477 20945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:44.367110 20945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b422a50d404bd2be3d06d94b31361e: Generated new TSK 0
I20260620 08:58:44.367964 20945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:44.371726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20863
2026-06-20T08:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:44 dist-test-slave-fb1v krb5kdc[20962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:44 dist-test-slave-fb1v krb5kdc[20962](info): set up 2 sockets
Jun 20 08:58:44 dist-test-slave-fb1v krb5kdc[20962](info): commencing operation
krb5kdc: starting...
W20260620 08:58:45.323472 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:45 dist-test-slave-fb1v krb5kdc[20962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:45.446924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:45.553134 20975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:45.553380 20975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:45.553419 20975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:45.557354 20975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:45.557411 20975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:45.557428 20975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:45.557443 20975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:45.557456 20975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:45.562624 20975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:45.563592 20975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:45.564567 20975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:45.572527 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:45.572528 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:45.572592 20981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:45.572657 20975 server_base.cc:1034] running on GCE node
I20260620 08:58:45.573287 20975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:45.574294 20975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:45.575457 20975 hybrid_clock.cc:648] HybridClock initialized: now 1781945925575443 us; error 27 us; skew 500 ppm
Jun 20 08:58:45 dist-test-slave-fb1v krb5kdc[20962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:45.578017 20975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:45.578811 20975 webserver.cc:460] Webserver started at http://127.0.0.1:41957/ using document root <none> and password file <none>
I20260620 08:58:45.579293 20975 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:45.579349 20975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:45.579530 20975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:45.581442 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7"
format_stamp: "Formatted at 2026-06-20 08:58:45 on dist-test-slave-fb1v"
server_key: "660d37cc8caae9db6a4de928856672a6"
server_key_iv: "d25502268f8930477f621114de5917cc"
server_key_version: "encryptionkey@0"
I20260620 08:58:45.581912 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7"
format_stamp: "Formatted at 2026-06-20 08:58:45 on dist-test-slave-fb1v"
server_key: "660d37cc8caae9db6a4de928856672a6"
server_key_iv: "d25502268f8930477f621114de5917cc"
server_key_version: "encryptionkey@0"
I20260620 08:58:45.585366 20975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:45.587566 20990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:45.588816 20975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:45.588961 20975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7"
format_stamp: "Formatted at 2026-06-20 08:58:45 on dist-test-slave-fb1v"
server_key: "660d37cc8caae9db6a4de928856672a6"
server_key_iv: "d25502268f8930477f621114de5917cc"
server_key_version: "encryptionkey@0"
I20260620 08:58:45.589058 20975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:45.611517 20975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:45.614940 20975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:45.615144 20975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:45.621989 20975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43055
I20260620 08:58:45.622009 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43055 every 8 connection(s)
I20260620 08:58:45.622880 20975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:45.625718 21043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:45.633019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 20975
I20260620 08:58:45.633000 21043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7: Bootstrap starting.
I20260620 08:58:45.633157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:45.633431 392 external_mini_cluster.cc:1428] Setting key 4c271de6a680c3f14067c302af4c588c
I20260620 08:58:45.635568 21043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:45.636384 21043 log.cc:824] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:45.638314 21043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7: No bootstrap required, opened a new log
Jun 20 08:58:45 dist-test-slave-fb1v krb5kdc[20962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:58:45.642506 21043 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260620 08:58:45.642805 21043 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:45.642874 21043 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7352340c37c4fa69c0b08f2bf5ffaf7, State: Initialized, Role: FOLLOWER
I20260620 08:58:45.643289 21043 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260620 08:58:45.643412 21043 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:45.643474 21043 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:45.643570 21043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:45.644424 21043 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260620 08:58:45.644761 21043 leader_election.cc:302] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7352340c37c4fa69c0b08f2bf5ffaf7; no voters:
I20260620 08:58:45.645052 21043 leader_election.cc:288] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:45.645149 21048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:45.645332 21048 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [term 1 LEADER]: Becoming Leader. State: Replica: b7352340c37c4fa69c0b08f2bf5ffaf7, State: Running, Role: LEADER
I20260620 08:58:45.645666 21048 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260620 08:58:45.646360 21043 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:45.647203 21049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } } }
I20260620 08:58:45.647336 21049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:45.647594 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7352340c37c4fa69c0b08f2bf5ffaf7. Latest consensus state: current_term: 1 leader_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7352340c37c4fa69c0b08f2bf5ffaf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } } }
I20260620 08:58:45.647778 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:45.648524 21057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:45.648978 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:45.634735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51375 (local address 127.0.0.1:43055)
0620 08:58:45.635136 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:45.635152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:45.635185 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:58:45.635884 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:45.635902 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:45.636245 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:45.636529 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:45.637356 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:45.638336 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:45.639457 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:45.639986 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:45.640127 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:45.642926 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:45.642931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:45.642945 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:45.642976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:45.645315 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:45.645820 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:45.645826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:45.645832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:45.645953 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:45.646228 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:45.646231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:45.646233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:45.646701 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:45.646873 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:45.647719 (+ 846us) server_negotiation.cc:299] Negotiation successful
0620 08:58:45.648028 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260620 08:58:45.651458 21057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:45.651960 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:45.656698 21057 catalog_manager.cc:1347] Generated new cluster ID: 9af0de8ab9ec4a04aadab167adfbf41e
I20260620 08:58:45.656826 21057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:45.666126 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:45.656912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58275 (local address 127.0.0.1:43055)
0620 08:58:45.657017 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:45.657021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:45.657426 (+ 405us) server_negotiation.cc:407] Connection header received
0620 08:58:45.657571 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:45.657574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:45.657624 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:45.657698 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:45.658478 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:45.659240 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:45.660533 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:45.661062 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:45.661178 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:45.662903 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:45.662907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:45.662910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:45.662934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:45.664483 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:45.664924 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:45.664927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:45.664929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:45.665002 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:45.665305 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:45.665309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:45.665311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:45.665527 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:45.665671 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:45.665857 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 08:58:45.665968 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:58:45.680629 21057 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:45.681643 21057 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:45.689477 21057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7352340c37c4fa69c0b08f2bf5ffaf7: Generated new TSK 0
I20260620 08:58:45.690277 21057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:45.697851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 20975
2026-06-20T08:58:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:45 dist-test-slave-fb1v krb5kdc[21074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:45 dist-test-slave-fb1v krb5kdc[21074](info): set up 2 sockets
Jun 20 08:58:45 dist-test-slave-fb1v krb5kdc[21074](info): commencing operation
krb5kdc: starting...
W20260620 08:58:46.668118 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:46 dist-test-slave-fb1v krb5kdc[21074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:46.795226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:46.913709 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:46.914001 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:46.914072 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:46.918365 21087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:46.918437 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:46.918465 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:46.918497 21087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:46.918525 21087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:46.923928 21087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:46.925045 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:46.926034 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:46.935294 21092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:46.935316 21095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:46.935388 21087 server_base.cc:1034] running on GCE node
W20260620 08:58:46.935288 21093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:46.936048 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:46.937074 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:46.938239 21087 hybrid_clock.cc:648] HybridClock initialized: now 1781945926938188 us; error 64 us; skew 500 ppm
Jun 20 08:58:46 dist-test-slave-fb1v krb5kdc[21074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:46.945264 21087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:46.946205 21087 webserver.cc:460] Webserver started at http://127.0.0.1:46229/ using document root <none> and password file <none>
I20260620 08:58:46.946760 21087 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:46.946821 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:46.947013 21087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:46.949110 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "8d63ea12999b4509a441ff59b86f728c"
format_stamp: "Formatted at 2026-06-20 08:58:46 on dist-test-slave-fb1v"
server_key: "348ae45910a853efd5646153111c8b38"
server_key_iv: "9dd1b3aa79f08df049330207ca0eeffe"
server_key_version: "encryptionkey@0"
I20260620 08:58:46.949635 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d63ea12999b4509a441ff59b86f728c"
format_stamp: "Formatted at 2026-06-20 08:58:46 on dist-test-slave-fb1v"
server_key: "348ae45910a853efd5646153111c8b38"
server_key_iv: "9dd1b3aa79f08df049330207ca0eeffe"
server_key_version: "encryptionkey@0"
I20260620 08:58:46.953413 21087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 08:58:46.955754 21102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:46.957103 21087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:46.957242 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "8d63ea12999b4509a441ff59b86f728c"
format_stamp: "Formatted at 2026-06-20 08:58:46 on dist-test-slave-fb1v"
server_key: "348ae45910a853efd5646153111c8b38"
server_key_iv: "9dd1b3aa79f08df049330207ca0eeffe"
server_key_version: "encryptionkey@0"
I20260620 08:58:46.957355 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:46.977353 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:46.980425 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:46.980585 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:46.988691 21087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38469
I20260620 08:58:46.988749 21154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38469 every 8 connection(s)
I20260620 08:58:46.989635 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:46.991709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21087
I20260620 08:58:46.991843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:46.992156 392 external_mini_cluster.cc:1428] Setting key 1ea0ce733a8279c5ff4e4b793b36a112
I20260620 08:58:46.992760 21155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:46.998600 21155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c: Bootstrap starting.
I20260620 08:58:47.000856 21155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:47.000880 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:46.993518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37853 (local address 127.0.0.1:38469)
0620 08:58:46.993948 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:46.993961 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:46.993995 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:58:46.994662 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:46.994677 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:46.995009 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:46.995314 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:46.995753 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:46.996707 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:46.997552 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:46.998011 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:46.998160 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:46.999228 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:46.999232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:46.999250 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:46.999645 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:46.999708 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:46.999732 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:46.999810 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:46.999948 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 08:58:47.000180 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
I20260620 08:58:47.001569 21155 log.cc:824] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:47.003363 21155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c: No bootstrap required, opened a new log
I20260620 08:58:47.003453 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:47.007895 21155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d63ea12999b4509a441ff59b86f728c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260620 08:58:47.008224 21155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:47.008292 21155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d63ea12999b4509a441ff59b86f728c, State: Initialized, Role: FOLLOWER
I20260620 08:58:47.009140 21155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d63ea12999b4509a441ff59b86f728c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260620 08:58:47.009441 21155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:47.009526 21155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:47.009629 21155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:47.010579 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:47.005833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58599 (local address 127.0.0.1:38469)
0620 08:58:47.005934 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:47.005946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:47.006049 (+ 103us) server_negotiation.cc:407] Connection header received
0620 08:58:47.006229 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:47.006232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:47.006275 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:47.006343 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:47.006659 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:47.007283 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:47.008569 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:47.008845 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:47.008940 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:47.010005 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:47.010009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:47.010011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:47.010023 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:47.010049 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:47.010060 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:47.010128 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:47.010306 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 08:58:47.010399 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:58:47.011108 21155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d63ea12999b4509a441ff59b86f728c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260620 08:58:47.011664 21155 leader_election.cc:302] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d63ea12999b4509a441ff59b86f728c; no voters:
I20260620 08:58:47.012287 21155 leader_election.cc:288] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:47.012414 21165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:47.012709 21165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [term 1 LEADER]: Becoming Leader. State: Replica: 8d63ea12999b4509a441ff59b86f728c, State: Running, Role: LEADER
I20260620 08:58:47.013134 21165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d63ea12999b4509a441ff59b86f728c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260620 08:58:47.014050 21155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:47.014999 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d63ea12999b4509a441ff59b86f728c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d63ea12999b4509a441ff59b86f728c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } } }
I20260620 08:58:47.015133 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:47.015540 21174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:47.015499 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d63ea12999b4509a441ff59b86f728c. Latest consensus state: current_term: 1 leader_uuid: "8d63ea12999b4509a441ff59b86f728c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d63ea12999b4509a441ff59b86f728c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } } }
I20260620 08:58:47.015688 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:47.018549 21174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:47.024905 21174 catalog_manager.cc:1347] Generated new cluster ID: ce46baf35eac4976bb036b0b1f0305df
I20260620 08:58:47.024986 21174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:47.040956 21174 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:47.041884 21174 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:47.048872 21174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d63ea12999b4509a441ff59b86f728c: Generated new TSK 0
I20260620 08:58:47.049564 21174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:47.060933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21087
2026-06-20T08:58:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:47 dist-test-slave-fb1v krb5kdc[21186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:47 dist-test-slave-fb1v krb5kdc[21186](info): set up 2 sockets
Jun 20 08:58:47 dist-test-slave-fb1v krb5kdc[21186](info): commencing operation
krb5kdc: starting...
W20260620 08:58:48.035567 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:48 dist-test-slave-fb1v krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:48.164011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:48.276901 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:48.277189 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:48.277230 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:48.281474 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:48.281529 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:48.281548 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:48.281563 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:48.281577 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:48.286854 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:48.287895 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:48.288856 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:48.297230 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:48.297325 21199 server_base.cc:1034] running on GCE node
W20260620 08:58:48.297227 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:48.297226 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:48.298017 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:48.299054 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:48.300221 21199 hybrid_clock.cc:648] HybridClock initialized: now 1781945928300198 us; error 36 us; skew 500 ppm
Jun 20 08:58:48 dist-test-slave-fb1v krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:48.307322 21199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:48.308248 21199 webserver.cc:460] Webserver started at http://127.0.0.1:34721/ using document root <none> and password file <none>
I20260620 08:58:48.308785 21199 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:48.308849 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:48.309027 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:48.311038 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685"
format_stamp: "Formatted at 2026-06-20 08:58:48 on dist-test-slave-fb1v"
server_key: "fe5e176365ac22dcd87a56eb34d7c3d3"
server_key_iv: "716dbe5d54caa4d1e90ac953b129dff1"
server_key_version: "encryptionkey@0"
I20260620 08:58:48.311758 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685"
format_stamp: "Formatted at 2026-06-20 08:58:48 on dist-test-slave-fb1v"
server_key: "fe5e176365ac22dcd87a56eb34d7c3d3"
server_key_iv: "716dbe5d54caa4d1e90ac953b129dff1"
server_key_version: "encryptionkey@0"
I20260620 08:58:48.315291 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:48.317803 21214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:48.319149 21199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:48.319278 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685"
format_stamp: "Formatted at 2026-06-20 08:58:48 on dist-test-slave-fb1v"
server_key: "fe5e176365ac22dcd87a56eb34d7c3d3"
server_key_iv: "716dbe5d54caa4d1e90ac953b129dff1"
server_key_version: "encryptionkey@0"
I20260620 08:58:48.319377 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:48.324100 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:48.328099 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:48.328289 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:48.335188 21199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33567
I20260620 08:58:48.335220 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33567 every 8 connection(s)
I20260620 08:58:48.336148 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:48.338984 21267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:48.341181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21199
I20260620 08:58:48.341312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:48.341553 392 external_mini_cluster.cc:1428] Setting key d4743d494f8608f6f2507cc11efde9f9
I20260620 08:58:48.344759 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685: Bootstrap starting.
I20260620 08:58:48.347029 21267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:48.347663 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:48.343325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40591 (local address 127.0.0.1:33567)
0620 08:58:48.343902 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:48.343923 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:48.343973 (+ 50us) server_negotiation.cc:407] Connection header received
0620 08:58:48.345001 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:48.345025 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:48.345328 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:48.345649 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:48.346416 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:48.346420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:48.346432 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:48.346442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:48.346495 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:48.346513 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:48.346608 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:48.346723 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 08:58:48.346876 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":135,"threads_started":1}
I20260620 08:58:48.347841 21267 log.cc:824] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:48.349756 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685: No bootstrap required, opened a new log
I20260620 08:58:48.350263 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:48.353647 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:48.352158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49921 (local address 127.0.0.1:33567)
0620 08:58:48.352275 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:48.352279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:48.352415 (+ 136us) server_negotiation.cc:407] Connection header received
0620 08:58:48.352521 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:48.352525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:48.352556 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:48.352635 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:48.353295 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:48.353298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:48.353301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:48.353312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:48.353337 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:48.353347 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:48.353415 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:48.353490 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:58:48.353526 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:58:48.353510 21267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33567 } }
I20260620 08:58:48.353783 21267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:48.353854 21267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9ad0f72f0b4f4abec8bcffd5eb5685, State: Initialized, Role: FOLLOWER
I20260620 08:58:48.354285 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33567 } }
I20260620 08:58:48.354427 21267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:48.354496 21267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:48.354589 21267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:48.355458 21267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33567 } }
I20260620 08:58:48.355798 21267 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9ad0f72f0b4f4abec8bcffd5eb5685; no voters:
I20260620 08:58:48.356076 21267 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:48.356165 21277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:48.356350 21277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9ad0f72f0b4f4abec8bcffd5eb5685, State: Running, Role: LEADER
I20260620 08:58:48.356691 21277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33567 } }
I20260620 08:58:48.357197 21267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:48.358194 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33567 } } }
I20260620 08:58:48.358224 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9ad0f72f0b4f4abec8bcffd5eb5685. Latest consensus state: current_term: 1 leader_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9ad0f72f0b4f4abec8bcffd5eb5685" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33567 } } }
I20260620 08:58:48.358399 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:48.358393 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:48.358821 21286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:48.362015 21286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:48.368036 21286 catalog_manager.cc:1347] Generated new cluster ID: 86e0b6d1d8834a1aa61b74fd9f86519b
I20260620 08:58:48.368103 21286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:48.374826 21286 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:48.375159 21286 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:48.379707 21286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9ad0f72f0b4f4abec8bcffd5eb5685: Generated new TSK 0
I20260620 08:58:48.380281 21286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:48.388236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21199
2026-06-20T08:58:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:48 dist-test-slave-fb1v krb5kdc[21298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:48 dist-test-slave-fb1v krb5kdc[21298](info): set up 2 sockets
Jun 20 08:58:48 dist-test-slave-fb1v krb5kdc[21298](info): commencing operation
krb5kdc: starting...
W20260620 08:58:49.379460 392 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:49 dist-test-slave-fb1v krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:49.494958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:49.599216 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:49.599457 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:49.599495 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:49.603369 21311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:49.603426 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:49.603443 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:49.603457 21311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:49.603471 21311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:49.608557 21311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:49.609504 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:49.610404 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:49.618290 21319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:49.618291 21316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:49.618320 21317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:49.618368 21311 server_base.cc:1034] running on GCE node
I20260620 08:58:49.619058 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:49.620148 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:49.621308 21311 hybrid_clock.cc:648] HybridClock initialized: now 1781945929621294 us; error 25 us; skew 500 ppm
Jun 20 08:58:49 dist-test-slave-fb1v krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:49.623996 21311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:49.624810 21311 webserver.cc:460] Webserver started at http://127.0.0.1:36153/ using document root <none> and password file <none>
I20260620 08:58:49.625311 21311 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:49.625372 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:49.625542 21311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:49.627466 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "07bc2c1d4c424314be7db137136e87ba"
format_stamp: "Formatted at 2026-06-20 08:58:49 on dist-test-slave-fb1v"
server_key: "9c9b0ab2499395a5686aa65a4462457e"
server_key_iv: "cf27a61f52ee0cecc8a5c24c5bd80d92"
server_key_version: "encryptionkey@0"
I20260620 08:58:49.628011 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "07bc2c1d4c424314be7db137136e87ba"
format_stamp: "Formatted at 2026-06-20 08:58:49 on dist-test-slave-fb1v"
server_key: "9c9b0ab2499395a5686aa65a4462457e"
server_key_iv: "cf27a61f52ee0cecc8a5c24c5bd80d92"
server_key_version: "encryptionkey@0"
I20260620 08:58:49.631420 21311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:49.633874 21326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:49.635042 21311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:49.635155 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "07bc2c1d4c424314be7db137136e87ba"
format_stamp: "Formatted at 2026-06-20 08:58:49 on dist-test-slave-fb1v"
server_key: "9c9b0ab2499395a5686aa65a4462457e"
server_key_iv: "cf27a61f52ee0cecc8a5c24c5bd80d92"
server_key_version: "encryptionkey@0"
I20260620 08:58:49.635246 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:49.655278 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:49.658265 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:49.658424 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:49.664769 21311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35871
I20260620 08:58:49.664808 21378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35871 every 8 connection(s)
I20260620 08:58:49.665647 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:49.668264 21379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:49.670325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21311
I20260620 08:58:49.670473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:49.670750 392 external_mini_cluster.cc:1428] Setting key b6b1209863b9bf8f42408c706e486f54
I20260620 08:58:49.674099 21379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba: Bootstrap starting.
I20260620 08:58:49.676541 21379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:49.677309 21379 log.cc:824] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:58:49 dist-test-slave-fb1v krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:58:49.679231 21379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba: No bootstrap required, opened a new log
I20260620 08:58:49.682771 21379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bc2c1d4c424314be7db137136e87ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260620 08:58:49.683003 21379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:49.683061 21379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bc2c1d4c424314be7db137136e87ba, State: Initialized, Role: FOLLOWER
I20260620 08:58:49.683549 21379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bc2c1d4c424314be7db137136e87ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260620 08:58:49.683770 21379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:49.683881 21379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:49.683991 21379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:49.684643 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:49.672082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:47207 (local address 127.0.0.1:35871)
0620 08:58:49.672478 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:49.672493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:49.672526 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:58:49.673313 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:49.673328 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:49.673653 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:49.673937 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:49.674683 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:49.675745 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:49.676579 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:49.677071 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:49.677225 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:49.679735 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:49.679740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:49.679753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:49.679778 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:49.681473 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:49.682156 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:49.682161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:49.682165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:49.682281 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:49.682620 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:49.682623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:49.682624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:49.683098 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:49.683289 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:49.683640 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 08:58:49.683890 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260620 08:58:49.684886 21379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bc2c1d4c424314be7db137136e87ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260620 08:58:49.685204 21379 leader_election.cc:302] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07bc2c1d4c424314be7db137136e87ba; no voters:
I20260620 08:58:49.685487 21379 leader_election.cc:288] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:49.685575 21384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:49.685815 21384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [term 1 LEADER]: Becoming Leader. State: Replica: 07bc2c1d4c424314be7db137136e87ba, State: Running, Role: LEADER
I20260620 08:58:49.686126 21384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bc2c1d4c424314be7db137136e87ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260620 08:58:49.686738 21379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:49.687234 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bc2c1d4c424314be7db137136e87ba. Latest consensus state: current_term: 1 leader_uuid: "07bc2c1d4c424314be7db137136e87ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bc2c1d4c424314be7db137136e87ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } } }
I20260620 08:58:49.687562 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:49.687919 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:49.687808 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07bc2c1d4c424314be7db137136e87ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bc2c1d4c424314be7db137136e87ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } } }
I20260620 08:58:49.688093 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:49.688319 21393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:49.691999 21393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:49.697415 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:49.689607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:38699 (local address 127.0.0.1:35871)
0620 08:58:49.689728 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:49.689732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:49.689938 (+ 206us) server_negotiation.cc:407] Connection header received
0620 08:58:49.690080 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:49.690082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:49.690133 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:49.690200 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:49.690988 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:49.691513 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:49.692730 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:49.692958 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:49.693042 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:49.694328 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:49.694331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:49.694333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:49.694354 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:49.695579 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:49.696090 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:49.696093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:49.696095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:49.696185 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:49.696535 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:49.696538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:49.696540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:49.696750 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:49.696935 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:49.697162 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 08:58:49.697264 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:58:49.698323 21393 catalog_manager.cc:1347] Generated new cluster ID: cfbc6c8389bb4469a9e3149e84cd2501
I20260620 08:58:49.698424 21393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:49.712746 21393 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:49.714032 21393 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:49.720733 21393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07bc2c1d4c424314be7db137136e87ba: Generated new TSK 0
I20260620 08:58:49.721408 21393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:49.725466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21311
W20260620 08:58:49.730104 21296 connection.cc:535] client connection to 127.0.0.1:35871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35871): Connection reset by peer (error 104)
2026-06-20T08:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:49 dist-test-slave-fb1v krb5kdc[21410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:49 dist-test-slave-fb1v krb5kdc[21410](info): set up 2 sockets
Jun 20 08:58:49 dist-test-slave-fb1v krb5kdc[21410](info): commencing operation
krb5kdc: starting...
W20260620 08:58:50.695557 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:50 dist-test-slave-fb1v krb5kdc[21410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:50.818693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:50.926784 21423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:50.927062 21423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:50.927104 21423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:50.931126 21423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:50.931193 21423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:50.931212 21423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:50.931228 21423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:50.931242 21423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:50.936604 21423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:50.937613 21423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:50.938571 21423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:50.946666 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:50.946698 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:50.946899 21423 server_base.cc:1034] running on GCE node
W20260620 08:58:50.946666 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:50.947561 21423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:50.948645 21423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:50.949816 21423 hybrid_clock.cc:648] HybridClock initialized: now 1781945930949792 us; error 35 us; skew 500 ppm
Jun 20 08:58:50 dist-test-slave-fb1v krb5kdc[21410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:50.952644 21423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:50.953498 21423 webserver.cc:460] Webserver started at http://127.0.0.1:43583/ using document root <none> and password file <none>
I20260620 08:58:50.954030 21423 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:50.954102 21423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:50.954267 21423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:50.956271 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "b0a36c96bcc1407abd963eb9ef20e6f9"
format_stamp: "Formatted at 2026-06-20 08:58:50 on dist-test-slave-fb1v"
server_key: "3e6a8994d8487b30b95f6dc697135fdf"
server_key_iv: "349cb842e7fdfbe14ab7e43594810f67"
server_key_version: "encryptionkey@0"
I20260620 08:58:50.956754 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0a36c96bcc1407abd963eb9ef20e6f9"
format_stamp: "Formatted at 2026-06-20 08:58:50 on dist-test-slave-fb1v"
server_key: "3e6a8994d8487b30b95f6dc697135fdf"
server_key_iv: "349cb842e7fdfbe14ab7e43594810f67"
server_key_version: "encryptionkey@0"
I20260620 08:58:50.960269 21423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:58:50.962536 21438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:50.963774 21423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:50.963881 21423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "b0a36c96bcc1407abd963eb9ef20e6f9"
format_stamp: "Formatted at 2026-06-20 08:58:50 on dist-test-slave-fb1v"
server_key: "3e6a8994d8487b30b95f6dc697135fdf"
server_key_iv: "349cb842e7fdfbe14ab7e43594810f67"
server_key_version: "encryptionkey@0"
I20260620 08:58:50.964051 21423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:50.984230 21423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:50.987298 21423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:50.987479 21423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:50.994161 21423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35071
I20260620 08:58:50.994189 21490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35071 every 8 connection(s)
I20260620 08:58:50.995141 21423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:50.997850 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:51.003238 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9: Bootstrap starting.
I20260620 08:58:51.005023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21423
I20260620 08:58:51.005167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:51.005436 392 external_mini_cluster.cc:1428] Setting key 1440a3bef262511a937547ecbd3975f5
I20260620 08:58:51.005918 21491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:51.006696 21491 log.cc:824] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:51.008963 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9: No bootstrap required, opened a new log
I20260620 08:58:51.013155 21491 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260620 08:58:51.013448 21491 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:51.013501 21491 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a36c96bcc1407abd963eb9ef20e6f9, State: Initialized, Role: FOLLOWER
I20260620 08:58:51.013984 21491 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260620 08:58:51.014142 21491 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:51.014211 21491 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:51.014307 21491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:51.015496 21491 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260620 08:58:51.015894 21491 leader_election.cc:302] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0a36c96bcc1407abd963eb9ef20e6f9; no voters:
I20260620 08:58:51.016274 21491 leader_election.cc:288] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:51.016295 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:51.006644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:60899 (local address 127.0.0.1:35071)
0620 08:58:51.007117 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:51.007133 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:51.007168 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:58:51.008041 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:51.008065 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:51.008486 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:51.008868 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:51.009276 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:51.010740 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:51.012315 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:51.013047 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:51.013220 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:51.014199 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:51.014205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:51.014222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:51.014732 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:51.014801 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:51.014832 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:51.014961 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:51.015169 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 08:58:51.015311 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":138,"threads_started":1}
I20260620 08:58:51.016434 21496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:51.016705 21496 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [term 1 LEADER]: Becoming Leader. State: Replica: b0a36c96bcc1407abd963eb9ef20e6f9, State: Running, Role: LEADER
I20260620 08:58:51.017021 21496 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260620 08:58:51.017532 21491 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:51.018177 21496 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a36c96bcc1407abd963eb9ef20e6f9. Latest consensus state: current_term: 1 leader_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } } }
I20260620 08:58:51.018898 21496 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:51.018774 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a36c96bcc1407abd963eb9ef20e6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } } }
I20260620 08:58:51.019299 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:51.019486 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 08:58:51.022794 21511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:58:51.022917 21511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:58:51.022992 21509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:51.023732 21509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:51.025473 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:51.021597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:57525 (local address 127.0.0.1:35071)
0620 08:58:51.021702 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:51.021707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:51.021842 (+ 135us) server_negotiation.cc:407] Connection header received
0620 08:58:51.021961 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:51.021964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:51.022009 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:51.022070 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:51.022308 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:51.022820 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:51.023659 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:51.023929 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:51.024016 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:51.025020 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:51.025024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:51.025027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:51.025039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:51.025067 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:51.025078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:51.025146 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:51.025286 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 08:58:51.025340 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:58:51.029776 21509 catalog_manager.cc:1347] Generated new cluster ID: 3afa43da3e33403995f79951fb19dcc6
I20260620 08:58:51.029850 21509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:51.049096 21509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:51.050410 21509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:51.060345 21509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0a36c96bcc1407abd963eb9ef20e6f9: Generated new TSK 0
I20260620 08:58:51.061148 21509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:51.069219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21423
W20260620 08:58:51.073808 21408 connection.cc:535] client connection to 127.0.0.1:35071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35071): Connection reset by peer (error 104)
2026-06-20T08:58:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:51 dist-test-slave-fb1v krb5kdc[21522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:51 dist-test-slave-fb1v krb5kdc[21522](info): set up 2 sockets
Jun 20 08:58:51 dist-test-slave-fb1v krb5kdc[21522](info): commencing operation
krb5kdc: starting...
W20260620 08:58:52.039783 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:52 dist-test-slave-fb1v krb5kdc[21522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:52.156653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:52.267684 21535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:52.267954 21535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:52.267998 21535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:52.272111 21535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:52.272177 21535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:52.272197 21535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:52.272212 21535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:52.272228 21535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:52.278980 21535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:52.280136 21535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:52.281119 21535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:52.289387 21543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:52.289400 21541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:52.289433 21535 server_base.cc:1034] running on GCE node
W20260620 08:58:52.289398 21540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:52.290148 21535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:52.291210 21535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:52.292377 21535 hybrid_clock.cc:648] HybridClock initialized: now 1781945932292350 us; error 38 us; skew 500 ppm
Jun 20 08:58:52 dist-test-slave-fb1v krb5kdc[21522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:52.295068 21535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:52.295993 21535 webserver.cc:460] Webserver started at http://127.0.0.1:37697/ using document root <none> and password file <none>
I20260620 08:58:52.296532 21535 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:52.296595 21535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:52.296792 21535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:52.298769 21535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "f55d210fe87d4f988a6b6d137110185e"
format_stamp: "Formatted at 2026-06-20 08:58:52 on dist-test-slave-fb1v"
server_key: "c3119c313423a1bf096d9be75f10225e"
server_key_iv: "7360a583d3c172cbfb041bc6fb69b628"
server_key_version: "encryptionkey@0"
I20260620 08:58:52.299301 21535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "f55d210fe87d4f988a6b6d137110185e"
format_stamp: "Formatted at 2026-06-20 08:58:52 on dist-test-slave-fb1v"
server_key: "c3119c313423a1bf096d9be75f10225e"
server_key_iv: "7360a583d3c172cbfb041bc6fb69b628"
server_key_version: "encryptionkey@0"
I20260620 08:58:52.302902 21535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:58:52.305850 21550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:52.307366 21535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:52.307498 21535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "f55d210fe87d4f988a6b6d137110185e"
format_stamp: "Formatted at 2026-06-20 08:58:52 on dist-test-slave-fb1v"
server_key: "c3119c313423a1bf096d9be75f10225e"
server_key_iv: "7360a583d3c172cbfb041bc6fb69b628"
server_key_version: "encryptionkey@0"
I20260620 08:58:52.307624 21535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:52.312234 21535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:52.315075 21535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:52.315212 21535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:52.322173 21535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43043
I20260620 08:58:52.322191 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43043 every 8 connection(s)
I20260620 08:58:52.323079 21535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:52.326536 21603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:52.332177 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e: Bootstrap starting.
I20260620 08:58:52.333104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21535
I20260620 08:58:52.333271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:52.333598 392 external_mini_cluster.cc:1428] Setting key e93bb61b1e098b952347b1cd753a0874
I20260620 08:58:52.335201 21603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:52.335989 21603 log.cc:824] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:52.338116 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e: No bootstrap required, opened a new log
I20260620 08:58:52.338693 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:52.334717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:42997 (local address 127.0.0.1:43043)
0620 08:58:52.335182 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:52.335196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:52.335231 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:58:52.335974 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:52.335992 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:52.336304 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:52.336681 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:52.337486 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:52.337491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:52.337503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:52.337514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:52.337566 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:52.337585 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:52.337671 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:52.337802 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 08:58:52.337952 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
I20260620 08:58:52.341408 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:52.342552 21603 raft_consensus.cc:348] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55d210fe87d4f988a6b6d137110185e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260620 08:58:52.342859 21603 raft_consensus.cc:374] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:52.342919 21603 raft_consensus.cc:729] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55d210fe87d4f988a6b6d137110185e, State: Initialized, Role: FOLLOWER
I20260620 08:58:52.343410 21603 consensus_queue.cc:260] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55d210fe87d4f988a6b6d137110185e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260620 08:58:52.343689 21603 raft_consensus.cc:388] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:52.343762 21603 raft_consensus.cc:482] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:52.343871 21603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:52.344949 21603 raft_consensus.cc:504] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55d210fe87d4f988a6b6d137110185e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260620 08:58:52.345259 21603 leader_election.cc:302] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f55d210fe87d4f988a6b6d137110185e; no voters:
I20260620 08:58:52.345626 21603 leader_election.cc:288] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:52.345772 21613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:52.346004 21613 raft_consensus.cc:686] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [term 1 LEADER]: Becoming Leader. State: Replica: f55d210fe87d4f988a6b6d137110185e, State: Running, Role: LEADER
I20260620 08:58:52.346295 21613 consensus_queue.cc:237] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55d210fe87d4f988a6b6d137110185e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260620 08:58:52.346769 21603 sys_catalog.cc:564] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:52.347918 21614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f55d210fe87d4f988a6b6d137110185e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55d210fe87d4f988a6b6d137110185e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } } }
I20260620 08:58:52.348043 21614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:52.348424 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f55d210fe87d4f988a6b6d137110185e. Latest consensus state: current_term: 1 leader_uuid: "f55d210fe87d4f988a6b6d137110185e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55d210fe87d4f988a6b6d137110185e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } } }
I20260620 08:58:52.348573 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:52.348435 21622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:52.350932 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:52.344181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:60551 (local address 127.0.0.1:43043)
0620 08:58:52.344299 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:52.344303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:52.349170 (+ 4867us) server_negotiation.cc:407] Connection header received
0620 08:58:52.349355 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:52.349358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:52.349399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:52.349494 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:52.350425 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:52.350429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:52.350432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:52.350449 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:52.350494 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:52.350510 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:52.350609 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:52.350708 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 08:58:52.350765 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:58:52.352154 21622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:52.357748 21622 catalog_manager.cc:1347] Generated new cluster ID: 5df92b0535d14d0a9e73d0834ac686e9
I20260620 08:58:52.357821 21622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:52.377072 21622 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:52.377432 21622 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:52.393900 21622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f55d210fe87d4f988a6b6d137110185e: Generated new TSK 0
I20260620 08:58:52.394644 21622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:52.400910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21535
2026-06-20T08:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:52 dist-test-slave-fb1v krb5kdc[21634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:52 dist-test-slave-fb1v krb5kdc[21634](info): set up 2 sockets
Jun 20 08:58:52 dist-test-slave-fb1v krb5kdc[21634](info): commencing operation
krb5kdc: starting...
W20260620 08:58:53.363572 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:53 dist-test-slave-fb1v krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:53.484651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:53.590624 21647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:53.590863 21647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:53.590902 21647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:53.594795 21647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:53.594849 21647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:53.594866 21647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:53.594882 21647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:53.594894 21647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:53.599975 21647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:53.600934 21647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:53.601882 21647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:53.609712 21653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:53.609754 21655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:53.609784 21652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:53.609874 21647 server_base.cc:1034] running on GCE node
I20260620 08:58:53.610482 21647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:53.611555 21647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:53.612767 21647 hybrid_clock.cc:648] HybridClock initialized: now 1781945933612755 us; error 25 us; skew 500 ppm
Jun 20 08:58:53 dist-test-slave-fb1v krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:53.618618 21647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:53.619622 21647 webserver.cc:460] Webserver started at http://127.0.0.1:36665/ using document root <none> and password file <none>
I20260620 08:58:53.620167 21647 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:53.620232 21647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:53.620422 21647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:53.622408 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "2dbb921dc8a9487986174ebd1bad57e0"
format_stamp: "Formatted at 2026-06-20 08:58:53 on dist-test-slave-fb1v"
server_key: "061ed1c5b61c63ec6ade959a82c00a7f"
server_key_iv: "e2038d1fd9b6ec1a2b680b440984e9ea"
server_key_version: "encryptionkey@0"
I20260620 08:58:53.622931 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dbb921dc8a9487986174ebd1bad57e0"
format_stamp: "Formatted at 2026-06-20 08:58:53 on dist-test-slave-fb1v"
server_key: "061ed1c5b61c63ec6ade959a82c00a7f"
server_key_iv: "e2038d1fd9b6ec1a2b680b440984e9ea"
server_key_version: "encryptionkey@0"
I20260620 08:58:53.626612 21647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:58:53.629180 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:53.630383 21647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:58:53.630492 21647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "2dbb921dc8a9487986174ebd1bad57e0"
format_stamp: "Formatted at 2026-06-20 08:58:53 on dist-test-slave-fb1v"
server_key: "061ed1c5b61c63ec6ade959a82c00a7f"
server_key_iv: "e2038d1fd9b6ec1a2b680b440984e9ea"
server_key_version: "encryptionkey@0"
I20260620 08:58:53.630589 21647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:53.649573 21647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:53.652532 21647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:53.652709 21647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:53.659534 21647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44257
I20260620 08:58:53.659933 21714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44257 every 8 connection(s)
I20260620 08:58:53.660492 21647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:53.661449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21647
I20260620 08:58:53.661595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:53.661891 392 external_mini_cluster.cc:1428] Setting key 2c34fbef9c3649c640f4bfb0a8ea2055
I20260620 08:58:53.663957 21715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:53.669994 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0: Bootstrap starting.
Jun 20 08:58:53 dist-test-slave-fb1v krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:58:53.673180 21715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:53.674053 21715 log.cc:824] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:53.676165 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0: No bootstrap required, opened a new log
I20260620 08:58:53.676740 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:53.663389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:46157 (local address 127.0.0.1:44257)
0620 08:58:53.663966 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:53.663980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:53.664017 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:58:53.664716 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:53.664732 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:53.665064 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:53.665384 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:53.666177 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:53.667183 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:53.668097 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:53.668573 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:53.668720 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:53.671819 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:53.671825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:53.671841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:53.671874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:53.673758 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:53.674312 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:53.674317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:53.674321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:53.674456 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:53.674817 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:53.674821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:53.674822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:53.675205 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:53.675360 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:53.675768 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 08:58:53.676006 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":188,"threads_started":1}
I20260620 08:58:53.679669 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:53.680078 21715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbb921dc8a9487986174ebd1bad57e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260620 08:58:53.680347 21715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:53.680402 21715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbb921dc8a9487986174ebd1bad57e0, State: Initialized, Role: FOLLOWER
I20260620 08:58:53.680871 21715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbb921dc8a9487986174ebd1bad57e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260620 08:58:53.681034 21715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:53.681105 21715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:53.681197 21715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:53.682291 21715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbb921dc8a9487986174ebd1bad57e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260620 08:58:53.682657 21715 leader_election.cc:302] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dbb921dc8a9487986174ebd1bad57e0; no voters:
I20260620 08:58:53.683004 21715 leader_election.cc:288] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:53.683100 21725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:53.683297 21725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2dbb921dc8a9487986174ebd1bad57e0, State: Running, Role: LEADER
I20260620 08:58:53.683696 21725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbb921dc8a9487986174ebd1bad57e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260620 08:58:53.684295 21715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:53.685189 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dbb921dc8a9487986174ebd1bad57e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbb921dc8a9487986174ebd1bad57e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260620 08:58:53.685315 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:53.685647 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dbb921dc8a9487986174ebd1bad57e0. Latest consensus state: current_term: 1 leader_uuid: "2dbb921dc8a9487986174ebd1bad57e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbb921dc8a9487986174ebd1bad57e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260620 08:58:53.686462 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0 [sys.catalog]: This master's current role is: LEADER
W20260620 08:58:53.689707 21741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:58:53.689837 21741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:58:53.689952 21734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:53.689965 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:53.681808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:52137 (local address 127.0.0.1:44257)
0620 08:58:53.681941 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:53.681944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:53.681989 (+ 45us) server_negotiation.cc:407] Connection header received
0620 08:58:53.682124 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:53.682126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:53.682164 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:53.682217 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:53.682986 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:53.683857 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:53.684919 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:53.685254 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:53.685365 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:53.687028 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:53.687031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:53.687034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:53.687058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:53.688351 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:53.688779 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:53.688782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:53.688783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:53.688848 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:53.689143 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:53.689146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:53.689148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:53.689311 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:53.689436 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:53.689704 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 08:58:53.689788 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:58:53.690788 21734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:53.697553 21734 catalog_manager.cc:1347] Generated new cluster ID: 271edac0f3544351be4eded4b3bae8a5
I20260620 08:58:53.697692 21734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:53.717981 21734 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:53.718941 21734 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:53.724097 21734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dbb921dc8a9487986174ebd1bad57e0: Generated new TSK 0
I20260620 08:58:53.724758 21734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:53.729702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21647
W20260620 08:58:53.734210 21632 connection.cc:535] client connection to 127.0.0.1:44257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44257): Connection reset by peer (error 104)
2026-06-20T08:58:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:53 dist-test-slave-fb1v krb5kdc[21746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:53 dist-test-slave-fb1v krb5kdc[21746](info): set up 2 sockets
Jun 20 08:58:53 dist-test-slave-fb1v krb5kdc[21746](info): commencing operation
krb5kdc: starting...
W20260620 08:58:54.695565 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:54 dist-test-slave-fb1v krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:54.822477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:54.929498 21759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:54.929765 21759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:54.929821 21759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:54.933799 21759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:54.933861 21759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:54.933887 21759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:54.934058 21759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:54.934118 21759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:54.939435 21759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:54.940476 21759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:54.941445 21759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:54.949591 21767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:54.949623 21765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:54.949651 21759 server_base.cc:1034] running on GCE node
W20260620 08:58:54.949776 21764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:54.950346 21759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:54.951464 21759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:54.952646 21759 hybrid_clock.cc:648] HybridClock initialized: now 1781945934952635 us; error 30 us; skew 500 ppm
Jun 20 08:58:54 dist-test-slave-fb1v krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:54.955371 21759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:54.956346 21759 webserver.cc:460] Webserver started at http://127.0.0.1:45065/ using document root <none> and password file <none>
I20260620 08:58:54.956887 21759 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:54.956959 21759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:54.957154 21759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:54.959194 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "ecbdf36af9144beab99a1a796270c79b"
format_stamp: "Formatted at 2026-06-20 08:58:54 on dist-test-slave-fb1v"
server_key: "487136f5f29171593930cfb8ffb5e1e4"
server_key_iv: "ff37db4452a107770cadb7ccca5e340e"
server_key_version: "encryptionkey@0"
I20260620 08:58:54.959789 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecbdf36af9144beab99a1a796270c79b"
format_stamp: "Formatted at 2026-06-20 08:58:54 on dist-test-slave-fb1v"
server_key: "487136f5f29171593930cfb8ffb5e1e4"
server_key_iv: "ff37db4452a107770cadb7ccca5e340e"
server_key_version: "encryptionkey@0"
I20260620 08:58:54.963410 21759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:54.965960 21774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:54.967142 21759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:58:54.967254 21759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "ecbdf36af9144beab99a1a796270c79b"
format_stamp: "Formatted at 2026-06-20 08:58:54 on dist-test-slave-fb1v"
server_key: "487136f5f29171593930cfb8ffb5e1e4"
server_key_iv: "ff37db4452a107770cadb7ccca5e340e"
server_key_version: "encryptionkey@0"
I20260620 08:58:54.967373 21759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:55.005795 21759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:55.009501 21759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:55.009675 21759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:55.019155 21759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44483
I20260620 08:58:55.019263 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44483 every 8 connection(s)
I20260620 08:58:55.020987 21759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:55.024184 21827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:55.029168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21759
I20260620 08:58:55.029311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:55.029560 392 external_mini_cluster.cc:1428] Setting key 625b1cdfd8bb5b73131ae592d59fcbce
I20260620 08:58:55.029979 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b: Bootstrap starting.
I20260620 08:58:55.032263 21827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:55.033059 21827 log.cc:824] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:55.035143 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b: No bootstrap required, opened a new log
I20260620 08:58:55.038616 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:55.030948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:44723 (local address 127.0.0.1:44483)
0620 08:58:55.031371 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:55.031387 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:55.031421 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:58:55.032070 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:55.032086 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:55.032470 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:55.032743 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:55.033068 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:55.034216 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:55.035376 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:55.035886 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:55.036001 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:55.036940 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:55.036946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:55.036957 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:55.037341 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:55.037394 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:55.037417 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:55.037518 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:55.037709 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 08:58:55.037843 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260620 08:58:55.038976 21827 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbdf36af9144beab99a1a796270c79b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260620 08:58:55.039196 21827 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:55.039249 21827 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbdf36af9144beab99a1a796270c79b, State: Initialized, Role: FOLLOWER
I20260620 08:58:55.039734 21827 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbdf36af9144beab99a1a796270c79b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260620 08:58:55.039871 21827 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:55.039935 21827 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:55.040026 21827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:55.041002 21827 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbdf36af9144beab99a1a796270c79b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260620 08:58:55.041313 21827 leader_election.cc:302] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecbdf36af9144beab99a1a796270c79b; no voters:
I20260620 08:58:55.041497 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:55.041622 21827 leader_election.cc:288] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:55.041733 21832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:55.041936 21832 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [term 1 LEADER]: Becoming Leader. State: Replica: ecbdf36af9144beab99a1a796270c79b, State: Running, Role: LEADER
I20260620 08:58:55.042315 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbdf36af9144beab99a1a796270c79b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260620 08:58:55.042945 21827 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:55.044299 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbdf36af9144beab99a1a796270c79b. Latest consensus state: current_term: 1 leader_uuid: "ecbdf36af9144beab99a1a796270c79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbdf36af9144beab99a1a796270c79b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } } }
I20260620 08:58:55.044438 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:55.044762 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecbdf36af9144beab99a1a796270c79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbdf36af9144beab99a1a796270c79b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } } }
I20260620 08:58:55.044817 21849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:55.044840 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:55.047889 21849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:55.048296 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:55.043636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:48061 (local address 127.0.0.1:44483)
0620 08:58:55.043749 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:55.043753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:55.043901 (+ 148us) server_negotiation.cc:407] Connection header received
0620 08:58:55.044083 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:55.044086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:55.044143 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:55.044225 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:55.044568 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:55.045320 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:55.046393 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:55.046724 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:55.046814 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:55.047811 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:55.047815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:55.047818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:55.047831 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:55.047858 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:55.047872 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:55.047965 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:55.048131 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 08:58:55.048167 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:58:55.053433 21849 catalog_manager.cc:1347] Generated new cluster ID: f9975e6824bd4e0584c3b779ddfa7dc6
I20260620 08:58:55.053503 21849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:55.060087 21849 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:55.061311 21849 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:55.065958 21849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecbdf36af9144beab99a1a796270c79b: Generated new TSK 0
I20260620 08:58:55.066685 21849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:55.074661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21759
W20260620 08:58:55.079823 21744 connection.cc:535] client connection to 127.0.0.1:44483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44483): Connection reset by peer (error 104)
2026-06-20T08:58:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:55 dist-test-slave-fb1v krb5kdc[21858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:55 dist-test-slave-fb1v krb5kdc[21858](info): set up 2 sockets
Jun 20 08:58:55 dist-test-slave-fb1v krb5kdc[21858](info): commencing operation
krb5kdc: starting...
W20260620 08:58:56.035573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:56 dist-test-slave-fb1v krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:58:56.159493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:56.271862 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:56.272122 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:56.272181 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:56.276201 21871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:56.276260 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:56.276286 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:56.276319 21871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:56.276341 21871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:56.281641 21871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:56.282666 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:56.283712 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:56.292213 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:56.292341 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:56.292359 21871 server_base.cc:1034] running on GCE node
W20260620 08:58:56.292443 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:56.293074 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:56.294195 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:56.295383 21871 hybrid_clock.cc:648] HybridClock initialized: now 1781945936295369 us; error 29 us; skew 500 ppm
Jun 20 08:58:56 dist-test-slave-fb1v krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:56.298101 21871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:58:56.298949 21871 webserver.cc:460] Webserver started at http://127.0.0.1:35541/ using document root <none> and password file <none>
I20260620 08:58:56.299470 21871 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:56.299533 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:56.299780 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:56.301741 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "74a2841aabbe4761960297efd7dd781f"
format_stamp: "Formatted at 2026-06-20 08:58:56 on dist-test-slave-fb1v"
server_key: "5a64a51ce69d5e945b3084592ab09464"
server_key_iv: "4bf32bb386479879a17e5c1aed840570"
server_key_version: "encryptionkey@0"
I20260620 08:58:56.302258 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "74a2841aabbe4761960297efd7dd781f"
format_stamp: "Formatted at 2026-06-20 08:58:56 on dist-test-slave-fb1v"
server_key: "5a64a51ce69d5e945b3084592ab09464"
server_key_iv: "4bf32bb386479879a17e5c1aed840570"
server_key_version: "encryptionkey@0"
I20260620 08:58:56.305874 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:58:56.308180 21886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:56.309358 21871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:58:56.309458 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "74a2841aabbe4761960297efd7dd781f"
format_stamp: "Formatted at 2026-06-20 08:58:56 on dist-test-slave-fb1v"
server_key: "5a64a51ce69d5e945b3084592ab09464"
server_key_iv: "4bf32bb386479879a17e5c1aed840570"
server_key_version: "encryptionkey@0"
I20260620 08:58:56.309582 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:56.313988 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:56.317556 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:56.317726 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:56.324940 21871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39403
I20260620 08:58:56.325080 21938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39403 every 8 connection(s)
I20260620 08:58:56.325893 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:56.328837 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:56.334178 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f: Bootstrap starting.
I20260620 08:58:56.335899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21871
I20260620 08:58:56.336031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:56.336305 392 external_mini_cluster.cc:1428] Setting key 704e8f36ccb774be711aae73009abe4e
I20260620 08:58:56.336791 21939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:56.337580 21939 log.cc:824] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:56.339483 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f: No bootstrap required, opened a new log
I20260620 08:58:56.341125 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:56.337377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:47153 (local address 127.0.0.1:39403)
0620 08:58:56.337830 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:56.337844 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:56.337877 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:58:56.338642 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:56.338661 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:56.338922 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:56.339218 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:56.340005 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:56.340010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:56.340020 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:56.340030 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:56.340081 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:56.340102 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:56.340196 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:56.340323 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 08:58:56.340470 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260620 08:58:56.343868 21939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a2841aabbe4761960297efd7dd781f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260620 08:58:56.344147 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:56.344166 21939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:56.344264 21939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a2841aabbe4761960297efd7dd781f, State: Initialized, Role: FOLLOWER
I20260620 08:58:56.344679 21939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a2841aabbe4761960297efd7dd781f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260620 08:58:56.344815 21939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:56.344883 21939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:56.344990 21939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:56.346594 21939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a2841aabbe4761960297efd7dd781f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260620 08:58:56.346956 21939 leader_election.cc:302] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a2841aabbe4761960297efd7dd781f; no voters:
I20260620 08:58:56.347306 21939 leader_election.cc:288] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:56.347759 21949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:56.347991 21949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [term 1 LEADER]: Becoming Leader. State: Replica: 74a2841aabbe4761960297efd7dd781f, State: Running, Role: LEADER
I20260620 08:58:56.348300 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:56.346522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:57939 (local address 127.0.0.1:39403)
0620 08:58:56.346694 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:56.346698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:56.346744 (+ 46us) server_negotiation.cc:407] Connection header received
0620 08:58:56.347025 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:56.347028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:56.347067 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:56.347162 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:56.347865 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:56.347868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:56.347871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:56.347883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:56.347910 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:56.347922 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:56.347979 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:56.348143 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 08:58:56.348182 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 08:58:56.348459 21949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a2841aabbe4761960297efd7dd781f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } }
I20260620 08:58:56.348690 21939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:56.349704 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74a2841aabbe4761960297efd7dd781f. Latest consensus state: current_term: 1 leader_uuid: "74a2841aabbe4761960297efd7dd781f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a2841aabbe4761960297efd7dd781f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } } }
I20260620 08:58:56.349818 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:56.350070 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74a2841aabbe4761960297efd7dd781f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a2841aabbe4761960297efd7dd781f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39403 } } }
I20260620 08:58:56.350171 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:56.350371 21958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:56.353461 21958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:56.359390 21958 catalog_manager.cc:1347] Generated new cluster ID: 72f5b7a6d19841aea1588956a909c314
I20260620 08:58:56.359460 21958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:56.378049 21958 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:56.378366 21958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:56.383122 21958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74a2841aabbe4761960297efd7dd781f: Generated new TSK 0
I20260620 08:58:56.383857 21958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:56.393925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21871
2026-06-20T08:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:56 dist-test-slave-fb1v krb5kdc[21970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:56 dist-test-slave-fb1v krb5kdc[21970](info): set up 2 sockets
Jun 20 08:58:56 dist-test-slave-fb1v krb5kdc[21970](info): commencing operation
krb5kdc: starting...
W20260620 08:58:57.367657 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:57 dist-test-slave-fb1v krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:58:57.488929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39843
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:57.593827 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:57.594069 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:57.594108 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:57.598052 21983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:57.598110 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:57.598129 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:57.598143 21983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:57.598157 21983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:57.603545 21983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:57.604519 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:57.605468 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:57.613729 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:57.613749 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:57.613741 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:57.614038 21983 server_base.cc:1034] running on GCE node
I20260620 08:58:57.614564 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:57.615535 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:57.616707 21983 hybrid_clock.cc:648] HybridClock initialized: now 1781945937616680 us; error 37 us; skew 500 ppm
Jun 20 08:58:57 dist-test-slave-fb1v krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:57.619290 21983 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:58:57.620195 21983 webserver.cc:460] Webserver started at http://127.0.98.62:38659/ using document root <none> and password file <none>
I20260620 08:58:57.620709 21983 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:57.620772 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:57.620962 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:57.622862 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "e4bac11894c3429c874c1be805c1f173"
format_stamp: "Formatted at 2026-06-20 08:58:57 on dist-test-slave-fb1v"
server_key: "518bc3d73af6aebf9454309e9bd17368"
server_key_iv: "e1876078c1df9d9c6208c0c1fcde2b50"
server_key_version: "encryptionkey@0"
I20260620 08:58:57.623387 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4bac11894c3429c874c1be805c1f173"
format_stamp: "Formatted at 2026-06-20 08:58:57 on dist-test-slave-fb1v"
server_key: "518bc3d73af6aebf9454309e9bd17368"
server_key_iv: "e1876078c1df9d9c6208c0c1fcde2b50"
server_key_version: "encryptionkey@0"
I20260620 08:58:57.626902 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:58:57.629272 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:57.630426 21983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:58:57.630535 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "e4bac11894c3429c874c1be805c1f173"
format_stamp: "Formatted at 2026-06-20 08:58:57 on dist-test-slave-fb1v"
server_key: "518bc3d73af6aebf9454309e9bd17368"
server_key_iv: "e1876078c1df9d9c6208c0c1fcde2b50"
server_key_version: "encryptionkey@0"
I20260620 08:58:57.630630 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:57.663708 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:57.666895 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:57.667086 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:57.673681 21983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39843
I20260620 08:58:57.673705 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39843 every 8 connection(s)
I20260620 08:58:57.674551 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:57.675215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 21983
I20260620 08:58:57.675343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:57.675668 392 external_mini_cluster.cc:1428] Setting key 7ba1e9fd10dc8495be7e1ab4b1fb5942
I20260620 08:58:57.677727 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:57.683068 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173: Bootstrap starting.
Jun 20 08:58:57 dist-test-slave-fb1v krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:58:57.685207 22051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:57.685854 22051 log.cc:824] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:57.687943 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173: No bootstrap required, opened a new log
I20260620 08:58:57.688864 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:57.676965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38995 (local address 127.0.98.62:39843)
0620 08:58:57.677379 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:57.677393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:57.677431 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:58:57.678099 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:57.678115 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:57.678428 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:57.678724 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:57.679488 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:57.680508 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:57.681321 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:57.681761 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:57.681900 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:57.684219 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:57.684224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:57.684235 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:57.684262 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:57.685776 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:57.686275 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:57.686280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:57.686284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:57.686387 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:57.686729 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:57.686733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:57.686734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:57.687186 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:57.687387 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:57.687736 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 08:58:57.687951 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260620 08:58:57.691876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:57.691937 22051 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4bac11894c3429c874c1be805c1f173" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260620 08:58:57.692174 22051 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:57.692229 22051 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bac11894c3429c874c1be805c1f173, State: Initialized, Role: FOLLOWER
I20260620 08:58:57.692605 22051 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4bac11894c3429c874c1be805c1f173" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260620 08:58:57.692714 22051 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:57.692767 22051 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:57.692855 22051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:57.693897 22051 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4bac11894c3429c874c1be805c1f173" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260620 08:58:57.694247 22051 leader_election.cc:302] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4bac11894c3429c874c1be805c1f173; no voters:
I20260620 08:58:57.694550 22051 leader_election.cc:288] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:57.694651 22061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:57.694854 22061 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [term 1 LEADER]: Becoming Leader. State: Replica: e4bac11894c3429c874c1be805c1f173, State: Running, Role: LEADER
I20260620 08:58:57.695217 22061 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4bac11894c3429c874c1be805c1f173" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260620 08:58:57.695786 22051 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:57.698271 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4bac11894c3429c874c1be805c1f173. Latest consensus state: current_term: 1 leader_uuid: "e4bac11894c3429c874c1be805c1f173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4bac11894c3429c874c1be805c1f173" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } } }
I20260620 08:58:57.698388 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:57.698417 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4bac11894c3429c874c1be805c1f173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4bac11894c3429c874c1be805c1f173" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } } }
I20260620 08:58:57.698513 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173 [sys.catalog]: This master's current role is: LEADER
W20260620 08:58:57.700769 22076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:58:57.700887 22076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:58:57.700974 22077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:57.701807 22077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:57.702315 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:57.693817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51641 (local address 127.0.98.62:39843)
0620 08:58:57.693930 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:57.693934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:57.694140 (+ 206us) server_negotiation.cc:407] Connection header received
0620 08:58:57.694231 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:57.694234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:57.694280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:57.694360 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:57.695078 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:57.695855 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:57.696945 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:57.697295 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:57.697391 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:57.699092 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:57.699096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:57.699099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:58:57.699121 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:57.700448 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:57.700879 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:57.700883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:57.700885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:57.700967 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:58:57.701469 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:58:57.701473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:58:57.701474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:58:57.701659 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:57.701786 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:57.702081 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:58:57.702167 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:58:57.707496 22077 catalog_manager.cc:1347] Generated new cluster ID: f0251d4b561f43569bc9d576fa515f53
I20260620 08:58:57.707567 22077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:57.725782 22077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:57.726739 22077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:57.731832 22077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4bac11894c3429c874c1be805c1f173: Generated new TSK 0
I20260620 08:58:57.732479 22077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:57.739319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 21983
W20260620 08:58:57.743880 21968 connection.cc:535] client connection to 127.0.98.62:39843 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39843): Connection reset by peer (error 104)
2026-06-20T08:58:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:57 dist-test-slave-fb1v krb5kdc[22082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:57 dist-test-slave-fb1v krb5kdc[22082](info): set up 2 sockets
Jun 20 08:58:57 dist-test-slave-fb1v krb5kdc[22082](info): commencing operation
krb5kdc: starting...
W20260620 08:58:58.703848 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:58:58 dist-test-slave-fb1v krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:58:58.823025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:58:58.930454 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:58:58.930747 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:58:58.930804 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:58:58.934861 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:58:58.934932 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:58:58.934959 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:58:58.934980 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:58:58.935001 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:58:58.940352 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:58:58.941452 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:58:58.942468 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:58:58.950968 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:58:58.950968 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:58.951153 22095 server_base.cc:1034] running on GCE node
W20260620 08:58:58.951090 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:58:58.951894 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:58:58.952941 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:58:58.954147 22095 hybrid_clock.cc:648] HybridClock initialized: now 1781945938954127 us; error 31 us; skew 500 ppm
Jun 20 08:58:58 dist-test-slave-fb1v krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:58:58.960121 22095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:58:58.961109 22095 webserver.cc:460] Webserver started at http://127.0.98.62:34357/ using document root <none> and password file <none>
I20260620 08:58:58.961663 22095 fs_manager.cc:359] Metadata directory not provided
I20260620 08:58:58.961737 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:58:58.961957 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:58:58.964154 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "363e99c6afa2445a98acef394768c6e4"
format_stamp: "Formatted at 2026-06-20 08:58:58 on dist-test-slave-fb1v"
server_key: "f2d35a068874ea7849507b76d4234eeb"
server_key_iv: "a7c9cc04042eeff097c1ec3ff9b022d8"
server_key_version: "encryptionkey@0"
I20260620 08:58:58.964720 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "363e99c6afa2445a98acef394768c6e4"
format_stamp: "Formatted at 2026-06-20 08:58:58 on dist-test-slave-fb1v"
server_key: "f2d35a068874ea7849507b76d4234eeb"
server_key_iv: "a7c9cc04042eeff097c1ec3ff9b022d8"
server_key_version: "encryptionkey@0"
I20260620 08:58:58.968621 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:58:58.971262 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:58:58.972673 22095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:58:58.972791 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "363e99c6afa2445a98acef394768c6e4"
format_stamp: "Formatted at 2026-06-20 08:58:58 on dist-test-slave-fb1v"
server_key: "f2d35a068874ea7849507b76d4234eeb"
server_key_iv: "a7c9cc04042eeff097c1ec3ff9b022d8"
server_key_version: "encryptionkey@0"
I20260620 08:58:58.972899 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:58:58.991859 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:58:58.995082 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:58:58.995276 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:58:59.002434 22095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46625
I20260620 08:58:59.002737 22162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46625 every 8 connection(s)
I20260620 08:58:59.003412 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:58:59.006213 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:58:59.009675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22095
I20260620 08:58:59.009815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:58:59.010106 392 external_mini_cluster.cc:1428] Setting key d8f9702ca25ec052637a515cfe0964c1
I20260620 08:58:59.012383 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4: Bootstrap starting.
I20260620 08:58:59.014705 22163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4: Neither blocks nor log segments found. Creating new log.
I20260620 08:58:59.015403 22163 log.cc:824] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:58:59.018312 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4: No bootstrap required, opened a new log
I20260620 08:58:59.020193 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:59.011528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57897 (local address 127.0.98.62:46625)
0620 08:58:59.012136 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:59.012152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:59.012186 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:58:59.012872 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:59.012890 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:59.013257 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:59.013529 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:59.013859 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:59.014829 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:59.015750 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:59.016379 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:59.016534 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:59.017815 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:59.017822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:59.017840 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:59.018381 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:59.018466 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:59.018508 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:59.018660 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:59.018936 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 08:58:59.019111 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":135,"threads_started":1}
I20260620 08:58:59.022708 22163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e99c6afa2445a98acef394768c6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46625 } }
I20260620 08:58:59.023000 22163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:58:59.023092 22163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363e99c6afa2445a98acef394768c6e4, State: Initialized, Role: FOLLOWER
I20260620 08:58:59.023569 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:58:59.023572 22163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e99c6afa2445a98acef394768c6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46625 } }
I20260620 08:58:59.023756 22163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:58:59.023821 22163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:58:59.023916 22163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:58:59.024933 22163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e99c6afa2445a98acef394768c6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46625 } }
I20260620 08:58:59.025272 22163 leader_election.cc:302] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 363e99c6afa2445a98acef394768c6e4; no voters:
I20260620 08:58:59.025629 22163 leader_election.cc:288] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:58:59.025720 22172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:58:59.025977 22172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [term 1 LEADER]: Becoming Leader. State: Replica: 363e99c6afa2445a98acef394768c6e4, State: Running, Role: LEADER
I20260620 08:58:59.026397 22172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e99c6afa2445a98acef394768c6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46625 } }
I20260620 08:58:59.027042 22163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:58:59.027460 22172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363e99c6afa2445a98acef394768c6e4. Latest consensus state: current_term: 1 leader_uuid: "363e99c6afa2445a98acef394768c6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e99c6afa2445a98acef394768c6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46625 } } }
I20260620 08:58:59.027650 22172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:59.027875 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "363e99c6afa2445a98acef394768c6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e99c6afa2445a98acef394768c6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46625 } } }
I20260620 08:58:59.027972 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:58:59.028079 22179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:58:59.030643 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:58:59.025691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57147 (local address 127.0.98.62:46625)
0620 08:58:59.025801 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 08:58:59.025806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:58:59.025930 (+ 124us) server_negotiation.cc:407] Connection header received
0620 08:58:59.026067 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:58:59.026070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:58:59.026119 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:58:59.026166 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:58:59.026562 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:59.027291 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:59.028438 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:58:59.028768 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:58:59.028844 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:58:59.030156 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:58:59.030160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:58:59.030163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:58:59.030176 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:58:59.030205 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:58:59.030218 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:58:59.030296 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:58:59.030469 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:58:59.030512 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:58:59.030997 22179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:58:59.036510 22179 catalog_manager.cc:1347] Generated new cluster ID: 6446a83d540b41e292a5cde72563e94e
I20260620 08:58:59.036586 22179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:58:59.045917 22179 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:58:59.046859 22179 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:58:59.062515 22179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 363e99c6afa2445a98acef394768c6e4: Generated new TSK 0
I20260620 08:58:59.063302 22179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:58:59.070534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22095
W20260620 08:58:59.075373 22080 connection.cc:535] client connection to 127.0.98.62:46625 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46625): Connection reset by peer (error 104)
2026-06-20T08:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:58:59 dist-test-slave-fb1v krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:58:59 dist-test-slave-fb1v krb5kdc[22194](info): set up 2 sockets
Jun 20 08:58:59 dist-test-slave-fb1v krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260620 08:59:00.039573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:00 dist-test-slave-fb1v krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:00.159718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36223
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:00.269001 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:00.269277 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:00.269336 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:00.273805 22207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:00.273895 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:00.273937 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:00.273965 22207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:00.273991 22207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:00.279528 22207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36223
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:00.281287 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:00.282845 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:00.291944 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:00.291990 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:00.291940 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:00.292248 22207 server_base.cc:1034] running on GCE node
I20260620 08:59:00.292748 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:00.293834 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:00.295181 22207 hybrid_clock.cc:648] HybridClock initialized: now 1781945940295140 us; error 55 us; skew 500 ppm
Jun 20 08:59:00 dist-test-slave-fb1v krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:00.298233 22207 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:00.299162 22207 webserver.cc:460] Webserver started at http://127.0.98.62:38769/ using document root <none> and password file <none>
I20260620 08:59:00.299785 22207 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:00.299865 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:00.300088 22207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:00.302109 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "7c54fbaa56a24bfbae858877ba42e34a"
format_stamp: "Formatted at 2026-06-20 08:59:00 on dist-test-slave-fb1v"
server_key: "c3793e28a60794a4eeec27d7c8077cb0"
server_key_iv: "a3ba662b522565e7ad86b9e97e237749"
server_key_version: "encryptionkey@0"
I20260620 08:59:00.302691 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c54fbaa56a24bfbae858877ba42e34a"
format_stamp: "Formatted at 2026-06-20 08:59:00 on dist-test-slave-fb1v"
server_key: "c3793e28a60794a4eeec27d7c8077cb0"
server_key_iv: "a3ba662b522565e7ad86b9e97e237749"
server_key_version: "encryptionkey@0"
I20260620 08:59:00.306519 22207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 08:59:00.309827 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:00.311339 22207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:59:00.311470 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "7c54fbaa56a24bfbae858877ba42e34a"
format_stamp: "Formatted at 2026-06-20 08:59:00 on dist-test-slave-fb1v"
server_key: "c3793e28a60794a4eeec27d7c8077cb0"
server_key_iv: "a3ba662b522565e7ad86b9e97e237749"
server_key_version: "encryptionkey@0"
I20260620 08:59:00.311583 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:00.316188 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:00.319478 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:00.319737 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:00.327515 22207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36223
I20260620 08:59:00.327554 22274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36223 every 8 connection(s)
I20260620 08:59:00.328441 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:00.331168 22275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:00.336027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22207
I20260620 08:59:00.336169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:00.336428 392 external_mini_cluster.cc:1428] Setting key e95314028c2dbe8ec4c60dfde22d569a
I20260620 08:59:00.336812 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a: Bootstrap starting.
I20260620 08:59:00.339205 22275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:00.340042 22275 log.cc:824] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:00.342224 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a: No bootstrap required, opened a new log
I20260620 08:59:00.342376 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:00.337799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37813 (local address 127.0.98.62:36223)
0620 08:59:00.338283 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:00.338297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:00.338330 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:59:00.339382 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:00.339406 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:00.339750 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:00.340089 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:00.341130 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:00.341135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:00.341146 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:59:00.341157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:00.341215 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:59:00.341236 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:00.341327 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:00.341467 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 08:59:00.341626 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260620 08:59:00.345387 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:59:00.346690 22275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54fbaa56a24bfbae858877ba42e34a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260620 08:59:00.346916 22275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:00.346971 22275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c54fbaa56a24bfbae858877ba42e34a, State: Initialized, Role: FOLLOWER
I20260620 08:59:00.347443 22275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54fbaa56a24bfbae858877ba42e34a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260620 08:59:00.347623 22275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:00.347692 22275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:00.347788 22275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:00.348675 22275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54fbaa56a24bfbae858877ba42e34a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260620 08:59:00.349017 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:00.347499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54193 (local address 127.0.98.62:36223)
0620 08:59:00.347633 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:00.347638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:00.347771 (+ 133us) server_negotiation.cc:407] Connection header received
0620 08:59:00.347873 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:00.347876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:00.347907 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:00.347978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:00.348618 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:00.348622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:00.348624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:59:00.348635 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:00.348662 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:59:00.348672 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:00.348740 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:00.348859 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 08:59:00.348894 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:59:00.349014 22275 leader_election.cc:302] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c54fbaa56a24bfbae858877ba42e34a; no voters:
I20260620 08:59:00.349362 22275 leader_election.cc:288] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:00.349431 22285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:00.349648 22285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [term 1 LEADER]: Becoming Leader. State: Replica: 7c54fbaa56a24bfbae858877ba42e34a, State: Running, Role: LEADER
I20260620 08:59:00.350018 22285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54fbaa56a24bfbae858877ba42e34a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260620 08:59:00.350591 22275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:00.350968 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c54fbaa56a24bfbae858877ba42e34a. Latest consensus state: current_term: 1 leader_uuid: "7c54fbaa56a24bfbae858877ba42e34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54fbaa56a24bfbae858877ba42e34a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } } }
I20260620 08:59:00.351076 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:00.351402 22289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:00.351547 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c54fbaa56a24bfbae858877ba42e34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54fbaa56a24bfbae858877ba42e34a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } } }
I20260620 08:59:00.351716 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:00.354442 22289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:00.360590 22289 catalog_manager.cc:1347] Generated new cluster ID: 6df53187bb7d425690bf2c4a8d382607
I20260620 08:59:00.360664 22289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:00.385990 22289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:00.386504 22289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:00.392711 22289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c54fbaa56a24bfbae858877ba42e34a: Generated new TSK 0
I20260620 08:59:00.393692 22289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:59:00.406926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22207
2026-06-20T08:59:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:00 dist-test-slave-fb1v krb5kdc[22306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:00 dist-test-slave-fb1v krb5kdc[22306](info): set up 2 sockets
Jun 20 08:59:00 dist-test-slave-fb1v krb5kdc[22306](info): commencing operation
krb5kdc: starting...
W20260620 08:59:01.379624 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:01 dist-test-slave-fb1v krb5kdc[22306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:01.496811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:01.610149 22319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:01.610457 22319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:01.610517 22319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:01.614847 22319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:01.614926 22319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:01.614957 22319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:01.614980 22319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:01.615105 22319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:01.620740 22319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:01.621829 22319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:01.622859 22319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:01.631083 22324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:01.631148 22319 server_base.cc:1034] running on GCE node
W20260620 08:59:01.631094 22327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:01.631094 22325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:01.631913 22319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:01.633319 22319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:01.634580 22319 hybrid_clock.cc:648] HybridClock initialized: now 1781945941634536 us; error 60 us; skew 500 ppm
Jun 20 08:59:01 dist-test-slave-fb1v krb5kdc[22306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:01.637360 22319 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:01.638271 22319 webserver.cc:460] Webserver started at http://127.0.98.62:41123/ using document root <none> and password file <none>
I20260620 08:59:01.638820 22319 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:01.638886 22319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:01.639076 22319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:01.641135 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "e89d44753a9a4a0c946044c131be7751"
format_stamp: "Formatted at 2026-06-20 08:59:01 on dist-test-slave-fb1v"
server_key: "1b4de4d97f10039a68ce1d271a149d2c"
server_key_iv: "137588ae6614eab3b3f4e75503115500"
server_key_version: "encryptionkey@0"
I20260620 08:59:01.641674 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "e89d44753a9a4a0c946044c131be7751"
format_stamp: "Formatted at 2026-06-20 08:59:01 on dist-test-slave-fb1v"
server_key: "1b4de4d97f10039a68ce1d271a149d2c"
server_key_iv: "137588ae6614eab3b3f4e75503115500"
server_key_version: "encryptionkey@0"
I20260620 08:59:01.645298 22319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 08:59:01.647750 22334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:01.648947 22319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:59:01.649062 22319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "e89d44753a9a4a0c946044c131be7751"
format_stamp: "Formatted at 2026-06-20 08:59:01 on dist-test-slave-fb1v"
server_key: "1b4de4d97f10039a68ce1d271a149d2c"
server_key_iv: "137588ae6614eab3b3f4e75503115500"
server_key_version: "encryptionkey@0"
I20260620 08:59:01.649168 22319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:01.669996 22319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:01.673053 22319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:01.673242 22319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:01.681430 22319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44289
I20260620 08:59:01.681617 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44289 every 8 connection(s)
I20260620 08:59:01.682538 22319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:01.683454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22319
I20260620 08:59:01.683593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:01.684063 392 external_mini_cluster.cc:1428] Setting key 3167cef3553a29b042e4370d303eb706
I20260620 08:59:01.685798 22387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:01.692425 22387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751: Bootstrap starting.
Jun 20 08:59:01 dist-test-slave-fb1v krb5kdc[22306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:01.695475 22387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:01.696779 22387 log.cc:824] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:01.699424 22387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751: No bootstrap required, opened a new log
I20260620 08:59:01.702100 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:01.685748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35439 (local address 127.0.98.62:44289)
0620 08:59:01.686334 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:01.686353 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:01.686397 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:59:01.687231 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:01.687252 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:01.687695 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:01.688058 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:01.688786 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:01.690261 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:01.691519 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:01.692241 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:01.692406 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:01.696047 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:01.696053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:01.696068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:01.696105 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:01.698445 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:01.699174 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:01.699181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:01.699187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:01.699385 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:01.699771 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:01.699774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:01.699776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:01.700324 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:01.700513 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:01.700942 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 08:59:01.701335 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":173,"threads_started":1}
I20260620 08:59:01.703701 22387 raft_consensus.cc:348] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89d44753a9a4a0c946044c131be7751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260620 08:59:01.703929 22387 raft_consensus.cc:374] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:01.704038 22387 raft_consensus.cc:729] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e89d44753a9a4a0c946044c131be7751, State: Initialized, Role: FOLLOWER
I20260620 08:59:01.704514 22387 consensus_queue.cc:260] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89d44753a9a4a0c946044c131be7751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260620 08:59:01.704648 22387 raft_consensus.cc:388] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:01.704802 22387 raft_consensus.cc:482] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:01.704948 22387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:01.705837 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:59:01.706039 22387 raft_consensus.cc:504] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89d44753a9a4a0c946044c131be7751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260620 08:59:01.706394 22387 leader_election.cc:302] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e89d44753a9a4a0c946044c131be7751; no voters:
I20260620 08:59:01.706697 22387 leader_election.cc:288] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:01.706799 22392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:01.706984 22392 raft_consensus.cc:686] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [term 1 LEADER]: Becoming Leader. State: Replica: e89d44753a9a4a0c946044c131be7751, State: Running, Role: LEADER
I20260620 08:59:01.707373 22392 consensus_queue.cc:237] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89d44753a9a4a0c946044c131be7751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260620 08:59:01.708079 22387 sys_catalog.cc:564] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:01.709111 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e89d44753a9a4a0c946044c131be7751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89d44753a9a4a0c946044c131be7751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260620 08:59:01.709239 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:01.709473 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e89d44753a9a4a0c946044c131be7751. Latest consensus state: current_term: 1 leader_uuid: "e89d44753a9a4a0c946044c131be7751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89d44753a9a4a0c946044c131be7751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260620 08:59:01.709564 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:01.709756 22406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:01.713523 22406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:01.719743 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:01.708942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57319 (local address 127.0.98.62:44289)
0620 08:59:01.709066 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:01.709071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:01.710451 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 08:59:01.710572 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:01.710575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:01.710629 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:01.710737 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:01.711670 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:01.712459 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:01.713987 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:01.714332 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:01.714453 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:01.716248 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:01.716252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:01.716255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:01.716281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:01.717930 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:01.718406 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:01.718410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:01.718412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:01.718503 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:01.718768 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:01.718772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:01.718779 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:01.719014 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:01.719146 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:01.719414 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 08:59:01.719550 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:59:01.721062 22406 catalog_manager.cc:1347] Generated new cluster ID: 1d703195063043d0a9ca7dc66ad4191f
I20260620 08:59:01.721127 22406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:01.739576 22406 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:01.741144 22406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:01.746376 22406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e89d44753a9a4a0c946044c131be7751: Generated new TSK 0
I20260620 08:59:01.747169 22406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:59:01.759478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22319
2026-06-20T08:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:01 dist-test-slave-fb1v krb5kdc[22418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:01 dist-test-slave-fb1v krb5kdc[22418](info): set up 2 sockets
Jun 20 08:59:01 dist-test-slave-fb1v krb5kdc[22418](info): commencing operation
krb5kdc: starting...
W20260620 08:59:02.727687 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:02 dist-test-slave-fb1v krb5kdc[22418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:02.847148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:02.956403 22431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:02.956698 22431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:02.956763 22431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:02.960839 22431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:02.960906 22431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:02.960934 22431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:02.960958 22431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:02.960987 22431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:02.966296 22431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:02.967295 22431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:02.968294 22431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:02.976989 22437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:02.977073 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:02.977119 22436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:02.977483 22431 server_base.cc:1034] running on GCE node
I20260620 08:59:02.978325 22431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:02.979588 22431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:02.980818 22431 hybrid_clock.cc:648] HybridClock initialized: now 1781945942980788 us; error 51 us; skew 500 ppm
Jun 20 08:59:02 dist-test-slave-fb1v krb5kdc[22418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:02.983549 22431 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:02.984524 22431 webserver.cc:460] Webserver started at http://127.0.98.62:40613/ using document root <none> and password file <none>
I20260620 08:59:02.985044 22431 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:02.985108 22431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:02.985311 22431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:02.987313 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "eaa3630171154f55b2ce13eaf771ae7e"
format_stamp: "Formatted at 2026-06-20 08:59:02 on dist-test-slave-fb1v"
server_key: "3b12c4b67c33dcfcc4fe0735b8c8db3c"
server_key_iv: "86ae6e3fafdf0722ca213d1615787b57"
server_key_version: "encryptionkey@0"
I20260620 08:59:02.987839 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "eaa3630171154f55b2ce13eaf771ae7e"
format_stamp: "Formatted at 2026-06-20 08:59:02 on dist-test-slave-fb1v"
server_key: "3b12c4b67c33dcfcc4fe0735b8c8db3c"
server_key_iv: "86ae6e3fafdf0722ca213d1615787b57"
server_key_version: "encryptionkey@0"
I20260620 08:59:02.991400 22431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 08:59:02.993940 22446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:02.995251 22431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 08:59:02.995378 22431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "eaa3630171154f55b2ce13eaf771ae7e"
format_stamp: "Formatted at 2026-06-20 08:59:02 on dist-test-slave-fb1v"
server_key: "3b12c4b67c33dcfcc4fe0735b8c8db3c"
server_key_iv: "86ae6e3fafdf0722ca213d1615787b57"
server_key_version: "encryptionkey@0"
I20260620 08:59:02.995481 22431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:03.022383 22431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:03.025478 22431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:03.025661 22431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:03.032349 22431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34603
I20260620 08:59:03.032373 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34603 every 8 connection(s)
I20260620 08:59:03.033322 22431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:03.033619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22431
I20260620 08:59:03.033756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:03.034039 392 external_mini_cluster.cc:1428] Setting key 1138ee9c5619f6d6eed42d1f92e2f116
I20260620 08:59:03.036255 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:03.042111 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e: Bootstrap starting.
I20260620 08:59:03.042971 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:03.035369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50267 (local address 127.0.98.62:34603)
0620 08:59:03.035785 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:03.035800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:03.035832 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:59:03.036476 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:03.036493 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:03.036827 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:03.037089 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:03.037436 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:03.038417 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:03.039573 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:03.040075 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:03.040206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:03.041045 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:03.041050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:03.041062 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:59:03.041437 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:03.041492 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:59:03.041515 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:03.041596 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:03.041743 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:59:03.041963 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260620 08:59:03.044333 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:03.045151 22499 log.cc:824] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:03.046504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:59:03.047119 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e: No bootstrap required, opened a new log
I20260620 08:59:03.050625 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa3630171154f55b2ce13eaf771ae7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34603 } }
I20260620 08:59:03.050859 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:03.050930 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa3630171154f55b2ce13eaf771ae7e, State: Initialized, Role: FOLLOWER
I20260620 08:59:03.051374 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa3630171154f55b2ce13eaf771ae7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34603 } }
I20260620 08:59:03.051534 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:03.051625 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:03.051733 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:03.052687 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa3630171154f55b2ce13eaf771ae7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34603 } }
I20260620 08:59:03.053043 22499 leader_election.cc:302] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa3630171154f55b2ce13eaf771ae7e; no voters:
I20260620 08:59:03.053361 22499 leader_election.cc:288] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:03.053442 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:03.053643 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [term 1 LEADER]: Becoming Leader. State: Replica: eaa3630171154f55b2ce13eaf771ae7e, State: Running, Role: LEADER
I20260620 08:59:03.054023 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa3630171154f55b2ce13eaf771ae7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34603 } }
I20260620 08:59:03.054546 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:03.055282 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:03.048980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47127 (local address 127.0.98.62:34603)
0620 08:59:03.049155 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:03.049161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:03.051191 (+ 2030us) server_negotiation.cc:407] Connection header received
0620 08:59:03.051287 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:03.051289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:03.051363 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:03.051437 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:03.051756 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:03.052306 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:03.053346 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:03.053668 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:03.053795 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:03.054813 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:03.054816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:03.054819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:59:03.054833 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:03.054861 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:59:03.054875 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:03.054953 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:03.055078 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 08:59:03.055148 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":67}
I20260620 08:59:03.055370 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa3630171154f55b2ce13eaf771ae7e. Latest consensus state: current_term: 1 leader_uuid: "eaa3630171154f55b2ce13eaf771ae7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa3630171154f55b2ce13eaf771ae7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34603 } } }
I20260620 08:59:03.055498 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:03.055711 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa3630171154f55b2ce13eaf771ae7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa3630171154f55b2ce13eaf771ae7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34603 } } }
I20260620 08:59:03.055867 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:03.056377 22518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:03.059455 22518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:03.065035 22518 catalog_manager.cc:1347] Generated new cluster ID: 1cc277b89a784645a91cce6a90fbd807
I20260620 08:59:03.065114 22518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:03.077670 22518 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:03.078987 22518 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:03.087235 22518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa3630171154f55b2ce13eaf771ae7e: Generated new TSK 0
I20260620 08:59:03.088055 22518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:59:03.099943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22431
2026-06-20T08:59:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:03 dist-test-slave-fb1v krb5kdc[22530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:03 dist-test-slave-fb1v krb5kdc[22530](info): set up 2 sockets
Jun 20 08:59:03 dist-test-slave-fb1v krb5kdc[22530](info): commencing operation
krb5kdc: starting...
W20260620 08:59:04.067739 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:04 dist-test-slave-fb1v krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:04.186501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:04.293321 22543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:04.293596 22543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:04.293653 22543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:04.297633 22543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:04.297699 22543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:04.297724 22543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:04.297745 22543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:04.297766 22543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:04.304677 22543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:04.306116 22543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:04.307163 22543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:04.315804 22551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:04.315821 22548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:04.315821 22549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:04.315871 22543 server_base.cc:1034] running on GCE node
I20260620 08:59:04.316881 22543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:04.318212 22543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:04.319394 22543 hybrid_clock.cc:648] HybridClock initialized: now 1781945944319372 us; error 33 us; skew 500 ppm
Jun 20 08:59:04 dist-test-slave-fb1v krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:04.325227 22543 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:04.326227 22543 webserver.cc:460] Webserver started at http://127.0.98.62:34913/ using document root <none> and password file <none>
I20260620 08:59:04.326776 22543 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:04.326845 22543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:04.327050 22543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:04.329221 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "69118c4ca7964c71a6afe7bf6faef257"
format_stamp: "Formatted at 2026-06-20 08:59:04 on dist-test-slave-fb1v"
server_key: "82368f47e3be1e3970b269d77a1a719e"
server_key_iv: "e317b281bac3f7384f80d366aab3ce72"
server_key_version: "encryptionkey@0"
I20260620 08:59:04.329735 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "69118c4ca7964c71a6afe7bf6faef257"
format_stamp: "Formatted at 2026-06-20 08:59:04 on dist-test-slave-fb1v"
server_key: "82368f47e3be1e3970b269d77a1a719e"
server_key_iv: "e317b281bac3f7384f80d366aab3ce72"
server_key_version: "encryptionkey@0"
I20260620 08:59:04.333283 22543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 08:59:04.335599 22558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:04.336782 22543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:59:04.336890 22543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "69118c4ca7964c71a6afe7bf6faef257"
format_stamp: "Formatted at 2026-06-20 08:59:04 on dist-test-slave-fb1v"
server_key: "82368f47e3be1e3970b269d77a1a719e"
server_key_iv: "e317b281bac3f7384f80d366aab3ce72"
server_key_version: "encryptionkey@0"
I20260620 08:59:04.337014 22543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:04.341475 22543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:04.344956 22543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:04.345101 22543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:04.351472 22543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32875
I20260620 08:59:04.351577 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32875 every 8 connection(s)
I20260620 08:59:04.352434 22543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:04.355075 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:04.360363 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257: Bootstrap starting.
I20260620 08:59:04.362613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22543
I20260620 08:59:04.362716 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:04.362761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:04.363031 392 external_mini_cluster.cc:1428] Setting key a81ca56dc99434135a9843fd50305bb4
I20260620 08:59:04.363519 22611 log.cc:824] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:04.365790 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257: No bootstrap required, opened a new log
I20260620 08:59:04.367692 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:04.364159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59235 (local address 127.0.98.62:32875)
0620 08:59:04.364580 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:04.364598 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:04.364637 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:59:04.365276 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:04.365291 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:04.365547 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:04.365838 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:04.366550 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:04.366554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:04.366566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:59:04.366576 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:04.366628 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:59:04.366645 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:04.366735 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:04.366859 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 08:59:04.367004 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260620 08:59:04.369411 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69118c4ca7964c71a6afe7bf6faef257" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260620 08:59:04.369664 22611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:04.369745 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69118c4ca7964c71a6afe7bf6faef257, State: Initialized, Role: FOLLOWER
I20260620 08:59:04.370177 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69118c4ca7964c71a6afe7bf6faef257" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260620 08:59:04.370333 22611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:04.370398 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:04.370466 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:59:04.370491 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:04.371524 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69118c4ca7964c71a6afe7bf6faef257" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260620 08:59:04.371897 22611 leader_election.cc:302] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69118c4ca7964c71a6afe7bf6faef257; no voters:
I20260620 08:59:04.372197 22611 leader_election.cc:288] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:04.372290 22620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:04.372448 22620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [term 1 LEADER]: Becoming Leader. State: Replica: 69118c4ca7964c71a6afe7bf6faef257, State: Running, Role: LEADER
I20260620 08:59:04.372812 22620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69118c4ca7964c71a6afe7bf6faef257" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260620 08:59:04.373453 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:04.373970 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69118c4ca7964c71a6afe7bf6faef257. Latest consensus state: current_term: 1 leader_uuid: "69118c4ca7964c71a6afe7bf6faef257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69118c4ca7964c71a6afe7bf6faef257" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } }
I20260620 08:59:04.374145 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:04.374402 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69118c4ca7964c71a6afe7bf6faef257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69118c4ca7964c71a6afe7bf6faef257" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } }
I20260620 08:59:04.374502 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:04.374778 22629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:04.377195 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:04.372555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56251 (local address 127.0.98.62:32875)
0620 08:59:04.372661 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:04.372666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:04.375622 (+ 2956us) server_negotiation.cc:407] Connection header received
0620 08:59:04.375784 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:04.375787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:04.375825 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:04.375911 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:04.376762 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:04.376766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:04.376769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:59:04.376780 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:04.376809 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:59:04.376822 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:04.376912 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:04.377041 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:59:04.377079 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:59:04.378211 22629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:04.383111 22629 catalog_manager.cc:1347] Generated new cluster ID: 81c32f8d31bb4dde8676b212a5cb8f82
I20260620 08:59:04.383179 22629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:04.399631 22629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:04.399955 22629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:04.404937 22629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69118c4ca7964c71a6afe7bf6faef257: Generated new TSK 0
I20260620 08:59:04.405572 22629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:59:04.411651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22543
2026-06-20T08:59:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1311 ms)
[----------] 18 tests from AuthTokenIssuingTest (23990 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:04 dist-test-slave-fb1v krb5kdc[22642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:04 dist-test-slave-fb1v krb5kdc[22642](info): set up 2 sockets
Jun 20 08:59:04 dist-test-slave-fb1v krb5kdc[22642](info): commencing operation
krb5kdc: starting...
W20260620 08:59:05.376273 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:05.496532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39715
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39715,127.0.98.61:46223,127.0.98.60:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:05.610744 22655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:59:05.611035 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:05.611058 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:05.611095 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:05.615155 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:05.615228 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:05.615247 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:05.615262 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:05.615276 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:05.620674 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39715,127.0.98.61:46223,127.0.98.60:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:05.621650 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:05.622622 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:05.630650 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:05.630659 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:05.630657 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:05.630851 22655 server_base.cc:1034] running on GCE node
I20260620 08:59:05.631429 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:05.632516 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:05.633690 22655 hybrid_clock.cc:648] HybridClock initialized: now 1781945945633675 us; error 29 us; skew 500 ppm
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:05.636358 22655 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:05.637187 22655 webserver.cc:460] Webserver started at http://127.0.98.62:37733/ using document root <none> and password file <none>
I20260620 08:59:05.637682 22655 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:05.637737 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:05.637893 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:05.639864 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "1f394d0bc2b041c8ba5f50b440c7f07d"
format_stamp: "Formatted at 2026-06-20 08:59:05 on dist-test-slave-fb1v"
server_key: "474d35a4f6762719d4a2df0e9f948096"
server_key_iv: "841f179cff4ce2f7d642abfb6cdf3ffc"
server_key_version: "encryptionkey@0"
I20260620 08:59:05.640357 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f394d0bc2b041c8ba5f50b440c7f07d"
format_stamp: "Formatted at 2026-06-20 08:59:05 on dist-test-slave-fb1v"
server_key: "474d35a4f6762719d4a2df0e9f948096"
server_key_iv: "841f179cff4ce2f7d642abfb6cdf3ffc"
server_key_version: "encryptionkey@0"
I20260620 08:59:05.643918 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:59:05.646216 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:05.647399 22655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:59:05.647501 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "1f394d0bc2b041c8ba5f50b440c7f07d"
format_stamp: "Formatted at 2026-06-20 08:59:05 on dist-test-slave-fb1v"
server_key: "474d35a4f6762719d4a2df0e9f948096"
server_key_iv: "841f179cff4ce2f7d642abfb6cdf3ffc"
server_key_version: "encryptionkey@0"
I20260620 08:59:05.647651 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:05.667984 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:05.671026 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:05.671226 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:05.678247 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39715
I20260620 08:59:05.678478 22722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39715 every 8 connection(s)
I20260620 08:59:05.679193 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:05.681918 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:05.682782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22655
I20260620 08:59:05.682885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:05.683145 392 external_mini_cluster.cc:1428] Setting key 6d671f8edc5c0d33fe88f524b5beaabc
I20260620 08:59:05.685802 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:05.703917 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.688289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39715 (local address 127.0.0.1:58894)
0620 08:59:05.688977 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.689054 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:05.690364 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:05.691424 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:05.691430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:05.691916 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:05.692165 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.692179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.693404 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.693406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.694280 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.694291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.694989 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.694993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.695153 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.696112 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:05.696139 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:05.698286 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:05.700344 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.700355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.700368 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.700747 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.701151 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.701154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.701156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.701298 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.702418 (+ 1120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:05.702424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:05.702573 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 08:59:05.702726 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 08:59:05.703012 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":1,"thread_start_us":154,"threads_started":1}
I20260620 08:59:05.703922 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.689426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.0.98.62:39715)
0620 08:59:05.689840 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:05.689853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:05.689909 (+ 56us) server_negotiation.cc:407] Connection header received
0620 08:59:05.690756 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:05.690761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:05.691162 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:05.691411 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:05.692325 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.693286 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.694420 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.694820 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.694964 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.698495 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:05.698502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:05.698522 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:05.698552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:05.700211 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.700914 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.700919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.700920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.701037 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.701503 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.701508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.701510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.702267 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:05.702408 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:05.702872 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 08:59:05.703107 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260620 08:59:05.706363 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:05.707154 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.706831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52336)
0620 08:59:05.706972 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.707067 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 08:59:05.707319 22673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
I20260620 08:59:05.708143 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.707928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52338)
0620 08:59:05.708028 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.708076 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 08:59:05.708268 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46223: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 08:59:05.710466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46223
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39715,127.0.98.61:46223,127.0.98.60:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:59:05.757506 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } attempt: 1
I20260620 08:59:05.758400 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.758062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52344)
0620 08:59:05.758236 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.758310 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 08:59:05.759075 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.758864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52346)
0620 08:59:05.758957 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.759009 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 08:59:05.759241 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46223: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
W20260620 08:59:05.818737 22730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:59:05.819013 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:05.819034 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:05.819085 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:05.823029 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:05.823096 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:05.823113 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:05.823128 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:05.823143 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:05.827471 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } attempt: 2
I20260620 08:59:05.828557 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.828168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52348)
0620 08:59:05.828388 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.828456 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 08:59:05.828745 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39715,127.0.98.61:46223,127.0.98.60:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:05.829309 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.829096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52352)
0620 08:59:05.829199 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.829248 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:59:05.829476 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46223: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46223: connect: Connection refused (error 111)
I20260620 08:59:05.829927 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:05.830912 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:05.838999 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:05.838982 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:05.838975 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:05.839430 22730 server_base.cc:1034] running on GCE node
I20260620 08:59:05.839994 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:05.841058 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:05.842229 22730 hybrid_clock.cc:648] HybridClock initialized: now 1781945945842212 us; error 33 us; skew 500 ppm
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:05.845321 22730 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 08:59:05.846626 22730 webserver.cc:460] Webserver started at http://127.0.98.61:39895/ using document root <none> and password file <none>
I20260620 08:59:05.847182 22730 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:05.847249 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:05.847438 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:05.849520 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data/instance:
uuid: "0f7f896ba9c44deab0a56d355d73f3bb"
format_stamp: "Formatted at 2026-06-20 08:59:05 on dist-test-slave-fb1v"
server_key: "6ff62a576947b4e2e30c4b10ff715152"
server_key_iv: "6a3608f80e1ee2c99466cc748c00151f"
server_key_version: "encryptionkey@0"
I20260620 08:59:05.850057 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal/instance:
uuid: "0f7f896ba9c44deab0a56d355d73f3bb"
format_stamp: "Formatted at 2026-06-20 08:59:05 on dist-test-slave-fb1v"
server_key: "6ff62a576947b4e2e30c4b10ff715152"
server_key_iv: "6a3608f80e1ee2c99466cc748c00151f"
server_key_version: "encryptionkey@0"
I20260620 08:59:05.853770 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:59:05.856271 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:05.857890 22730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:59:05.858000 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal
uuid: "0f7f896ba9c44deab0a56d355d73f3bb"
format_stamp: "Formatted at 2026-06-20 08:59:05 on dist-test-slave-fb1v"
server_key: "6ff62a576947b4e2e30c4b10ff715152"
server_key_iv: "6a3608f80e1ee2c99466cc748c00151f"
server_key_version: "encryptionkey@0"
I20260620 08:59:05.858139 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:05.880579 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:05.883643 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:05.883838 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:05.890789 22730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46223
I20260620 08:59:05.891039 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46223 every 8 connection(s)
I20260620 08:59:05.891772 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
I20260620 08:59:05.894428 22798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:05.896734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22730
I20260620 08:59:05.896873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-1/wal/instance
I20260620 08:59:05.897179 392 external_mini_cluster.cc:1428] Setting key 45dc007d436d9ec8c926613ad55b7b78
I20260620 08:59:05.898144 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:05.913180 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.900090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.0.98.62:39715)
0620 08:59:05.900324 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:05.900327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:05.900913 (+ 586us) server_negotiation.cc:407] Connection header received
0620 08:59:05.901819 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:05.901823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:05.901879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:05.901952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:05.903084 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.903639 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.904765 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.905204 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.905389 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.908964 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:05.908968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:05.908971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:05.909003 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:05.910442 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.911052 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.911055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.911057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.911138 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.911594 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.911601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.911603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.912083 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:05.912181 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:05.912914 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 08:59:05.913033 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 08:59:05.914290 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.900320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39715 (local address 127.0.0.1:58896)
0620 08:59:05.900751 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.900794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:05.901621 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:05.902112 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:05.902121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:05.902484 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:05.902944 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.902955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.903772 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.903799 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.904640 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.904650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.905519 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.905526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.905743 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.906683 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:05.906704 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:05.908770 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:05.910571 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.910578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.910595 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.910942 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.911223 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.911226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.911228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.911403 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.912206 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:05.912213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:05.912581 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 08:59:05.912827 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 08:59:05.913204 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260620 08:59:05.915771 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:05.920662 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } attempt: 3
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 08:59:05.924793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36101
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39715,127.0.98.61:46223,127.0.98.60:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:59:05.924978 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.916228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52354)
0620 08:59:05.916350 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.916364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:05.916454 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:05.917041 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:05.917044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:05.917200 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:05.917285 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.917290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.918331 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.918334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.918841 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.918848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.919387 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.919390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.919557 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.920217 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:05.920228 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:05.921642 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:05.923396 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.923399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.923401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.923571 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.923935 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.923938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.923940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.924023 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.924482 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:05.924486 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:05.924580 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 08:59:05.924687 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:59:05.924814 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":57}
I20260620 08:59:05.925325 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.916389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.0.98.61:46223)
0620 08:59:05.916736 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:05.916744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:05.916760 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:59:05.916788 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:05.916792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:05.916932 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:05.917042 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:05.917381 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.918238 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.918936 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.919292 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.919379 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.921752 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:05.921761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:05.921775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:05.921794 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:05.923299 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.923733 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.923739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.923740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.923842 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.924105 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.924107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.924109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.924372 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:05.924504 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:05.924775 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 08:59:05.924911 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":272,"thread_start_us":185,"threads_started":1}
Jun 20 08:59:05 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:59:05.927300 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:05.928053 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.927790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52638)
0620 08:59:05.927903 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.927983 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:59:05.928241 22748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
I20260620 08:59:05.929046 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.928849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52640)
0620 08:59:05.928936 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.928982 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 08:59:05.929199 22798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36101: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
I20260620 08:59:05.930347 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.921189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.0.98.61:46223)
0620 08:59:05.921640 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:05.921647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:05.921659 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:59:05.921687 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:05.921689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:05.921720 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:05.921816 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:05.922232 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.922842 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.923680 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.923952 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.924046 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.926910 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:05.926914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:05.926917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:05.926941 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:05.928753 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.929198 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.929201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.929203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.929294 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:05.929609 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:05.929611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:05.929612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:05.929802 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:05.929871 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:05.930140 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 08:59:05.930226 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":96,"threads_started":1}
I20260620 08:59:05.930428 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.921096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52366)
0620 08:59:05.921294 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.921308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:05.921387 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:05.921848 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:05.921850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:05.922035 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:05.922128 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.922134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.922954 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.922957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.923529 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:05.923538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:05.924053 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:05.924056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:05.924157 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:05.925353 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:05.925368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:05.926778 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:05.928841 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.928844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.928846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.929090 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.929413 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:05.929417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:05.929419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:05.929515 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:05.929883 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:05.929886 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:05.929993 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 08:59:05.930111 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 08:59:05.930263 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 08:59:05.931062 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:05.931711 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.931472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52642)
0620 08:59:05.931572 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.931643 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:59:05.932291 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.932071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52654)
0620 08:59:05.932190 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.932234 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 08:59:05.932397 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36101: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
I20260620 08:59:05.963553 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } attempt: 1
I20260620 08:59:05.964351 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.964082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52660)
0620 08:59:05.964219 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.964277 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 08:59:05.964939 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.964753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52664)
0620 08:59:05.964846 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.964887 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:59:05.965040 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36101: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
I20260620 08:59:05.978493 22798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } attempt: 1
I20260620 08:59:05.979400 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.979104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52676)
0620 08:59:05.979257 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.979321 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 08:59:05.980224 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:05.979940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52682)
0620 08:59:05.980089 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:05.980161 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 08:59:05.980437 22798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36101: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
W20260620 08:59:06.037541 22805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:59:06.037853 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:06.037878 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:06.037915 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 08:59:06.040211 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } attempt: 2
I20260620 08:59:06.041016 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.040718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52686)
0620 08:59:06.040882 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.040942 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 08:59:06.041683 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.041486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52696)
0620 08:59:06.041579 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.041622 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:59:06.041790 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36101: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
W20260620 08:59:06.042182 22805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:06.042233 22805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:06.042263 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:06.042294 22805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:06.042322 22805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:06.047814 22805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39715,127.0.98.61:46223,127.0.98.60:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:06.048599 22798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } attempt: 2
I20260620 08:59:06.048882 22805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:06.049304 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.049048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52712)
0620 08:59:06.049176 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.049235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 08:59:06.049912 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.049726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52728)
0620 08:59:06.049818 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.049861 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:59:06.049976 22805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:06.050014 22798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36101: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36101: connect: Connection refused (error 111)
W20260620 08:59:06.058748 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:06.058748 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:06.058949 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:06.059149 22805 server_base.cc:1034] running on GCE node
I20260620 08:59:06.059717 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:06.060812 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:06.062086 22805 hybrid_clock.cc:648] HybridClock initialized: now 1781945946062058 us; error 75 us; skew 500 ppm
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:06.064857 22805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 08:59:06.065721 22805 webserver.cc:460] Webserver started at http://127.0.98.60:41701/ using document root <none> and password file <none>
I20260620 08:59:06.066236 22805 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:06.066299 22805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:06.066493 22805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:06.068521 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data/instance:
uuid: "4f5fe4b62ca24aa4be6a1212392873a9"
format_stamp: "Formatted at 2026-06-20 08:59:06 on dist-test-slave-fb1v"
server_key: "b311eaf8c52e3573c6ade7d121b914ef"
server_key_iv: "646e54820dbc671cf74c5bcb38d8c8e0"
server_key_version: "encryptionkey@0"
I20260620 08:59:06.069046 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal/instance:
uuid: "4f5fe4b62ca24aa4be6a1212392873a9"
format_stamp: "Formatted at 2026-06-20 08:59:06 on dist-test-slave-fb1v"
server_key: "b311eaf8c52e3573c6ade7d121b914ef"
server_key_iv: "646e54820dbc671cf74c5bcb38d8c8e0"
server_key_version: "encryptionkey@0"
I20260620 08:59:06.072719 22805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 08:59:06.075317 22821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:06.076790 22805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 08:59:06.077078 22805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal
uuid: "4f5fe4b62ca24aa4be6a1212392873a9"
format_stamp: "Formatted at 2026-06-20 08:59:06 on dist-test-slave-fb1v"
server_key: "b311eaf8c52e3573c6ade7d121b914ef"
server_key_iv: "646e54820dbc671cf74c5bcb38d8c8e0"
server_key_version: "encryptionkey@0"
I20260620 08:59:06.077287 22805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:06.096706 22805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:06.099807 22805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:06.099999 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:06.107384 22805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36101
I20260620 08:59:06.107690 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36101 every 8 connection(s)
I20260620 08:59:06.108367 22805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
I20260620 08:59:06.111934 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:06.112914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 22805
I20260620 08:59:06.113023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945671001899-392-0/minicluster-data/master-2/wal/instance
I20260620 08:59:06.113361 392 external_mini_cluster.cc:1428] Setting key 993bc0d2ef041f59ec87cdfb0b933ec5
I20260620 08:59:06.117403 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:06.129393 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.119461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.0.98.62:39715)
0620 08:59:06.119593 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.119598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.120288 (+ 690us) server_negotiation.cc:407] Connection header received
0620 08:59:06.120388 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.120392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.120444 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.120514 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.121283 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.121914 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.122814 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.123017 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.123103 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.125639 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.125644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.125647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.125688 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.127728 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.128158 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.128161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.128163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.128237 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.128508 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.128511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.128513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.128723 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.128823 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.129140 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.129257 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:59:06.129463 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.119358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39715 (local address 127.0.0.1:58908)
0620 08:59:06.120160 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.120208 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:06.120305 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:06.120567 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:06.120572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:06.120995 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:06.121146 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.121153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.122060 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.122070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.122685 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.122694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.123158 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.123162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.123251 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.123740 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:06.123771 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:06.125398 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:06.127834 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.127842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.127847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.128061 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.128315 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.128319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.128320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.128409 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.128830 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:06.128835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:06.128936 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 08:59:06.129047 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:59:06.129193 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":160,"threads_started":1}
I20260620 08:59:06.129465 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.114920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.0.98.60:36101)
0620 08:59:06.115374 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.115395 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.115441 (+ 46us) server_negotiation.cc:407] Connection header received
0620 08:59:06.116359 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.116383 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.116814 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.117205 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.117883 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.119406 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.120398 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.121053 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.121212 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.123523 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.123528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.123537 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.123572 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.125416 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.126956 (+ 1540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.126961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.126966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.127077 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.127326 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.127329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.127331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.127781 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.127970 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.128247 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.128527 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":108,"threads_started":1}
I20260620 08:59:06.130750 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:06.140931 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } attempt: 3
I20260620 08:59:06.141139 22798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } attempt: 3
I20260620 08:59:06.143460 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.131416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.0.98.61:46223)
0620 08:59:06.131535 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.131540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.131553 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:59:06.131594 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.131597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.131660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.131733 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.132138 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.132907 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.133917 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.134261 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.134360 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.136863 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.136868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.136870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.136899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.141169 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.142049 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.142053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.142055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.142151 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.142477 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.142481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.142483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.142717 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.142784 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.143121 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.143262 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:59:06.143970 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.131289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46223 (local address 127.0.0.1:52376)
0620 08:59:06.131396 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.131410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:06.131480 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:06.131761 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:06.131764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:06.131897 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:06.132012 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.132019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.133019 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.133023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.133808 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.133818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.134487 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.134491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.134595 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.135156 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:06.135169 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:06.136699 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:06.141309 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.141313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.141315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.141559 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.142277 (+ 718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.142281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.142282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.142372 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.142855 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:06.142858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:06.142948 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 08:59:06.143664 (+ 716us) client_negotiation.cc:240] Negotiation successful
0620 08:59:06.143816 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:59:06.144443 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.131966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.98.62:39715)
0620 08:59:06.132073 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.132077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.132089 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:59:06.132157 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.132159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.132193 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.132247 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.132915 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.133624 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.134625 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.134915 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.135011 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.137665 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.137669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.137671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.137696 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.142017 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.142491 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.142495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.142496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.142582 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.142878 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.142882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.142884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.143112 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.143313 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.143943 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.144292 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:59:06.144622 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 08:59:06.151499 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.141502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52744)
0620 08:59:06.141790 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.141808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:06.141870 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:06.142533 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:06.142536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:06.142694 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:06.142784 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.142790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.144466 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.144469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.145140 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.145150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.146011 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.146015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.146098 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.146515 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:06.146529 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:06.147803 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:06.149860 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.149863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.149865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.150115 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.150491 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.150493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.150495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.150568 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.151026 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:06.151029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:06.151119 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 08:59:06.151200 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:59:06.151333 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260620 08:59:06.151499 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.141498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52746)
0620 08:59:06.141674 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.141687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:06.141763 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:06.142566 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:06.142569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:06.142730 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:06.142837 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.142844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.143939 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.143942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.144731 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.144741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.145276 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.145279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.145357 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.146004 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:06.146018 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:06.147919 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:06.149924 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.149927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.149929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.150133 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.150474 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.150478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.150479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.150564 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.151026 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:06.151029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:06.151120 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 08:59:06.151205 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 08:59:06.151328 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 08:59:06.152177 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.141849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.0.98.60:36101)
0620 08:59:06.142351 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.142355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.142368 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:59:06.142400 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.142403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.142443 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.142931 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.143014 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.143833 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.144867 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.145183 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.145272 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.148032 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.148036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.148038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.148058 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.149824 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.150261 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.150264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.150266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.150371 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.150671 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.150673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.150675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.150881 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.151807 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.151882 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.152014 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":428,"spinlock_wait_cycles":1280,"thread_start_us":110,"threads_started":1}
I20260620 08:59:06.153021 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.141747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.98.60:36101)
0620 08:59:06.142324 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.142333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.142343 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:59:06.142372 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.142375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.142414 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.142871 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.142946 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.144203 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.145280 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.145894 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.147070 (+ 1176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.147924 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.147928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.147930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.147952 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.149739 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.150261 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.150264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.150266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.150399 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.150764 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.150766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.150768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.150926 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.152113 (+ 1187us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.152180 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.152872 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":500,"spinlock_wait_cycles":3456}
Jun 20 08:59:06 dist-test-slave-fb1v krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:59:06.154263 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.145136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36101 (local address 127.0.0.1:52756)
0620 08:59:06.145703 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:06.145716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:06.145788 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:06.146086 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:06.146089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:06.146247 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:06.146327 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.146333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.147259 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.147262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.147911 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:06.147919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.149203 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.149207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:06.149278 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.149665 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:06.149678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:06.151012 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:06.152899 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.152902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.152904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.153112 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.153387 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:06.153389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:06.153391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:06.153462 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:06.153830 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:06.153833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:06.153909 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 08:59:06.154010 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 08:59:06.154120 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494}
I20260620 08:59:06.154774 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.145315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.0.98.60:36101)
0620 08:59:06.145552 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.145568 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.145853 (+ 285us) server_negotiation.cc:407] Connection header received
0620 08:59:06.145877 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.145879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.145911 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.145977 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.146433 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.147155 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.148027 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.149005 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.149098 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.151117 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.151120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.151122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.151140 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.152781 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.153217 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.153220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.153221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.153303 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.153540 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.153543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.153545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.153719 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.153799 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.154510 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.154620 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260620 08:59:06.155565 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: Bootstrap starting.
I20260620 08:59:06.155630 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: Bootstrap starting.
I20260620 08:59:06.157411 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: Bootstrap starting.
I20260620 08:59:06.158330 22798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:06.158330 22723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:06.159284 22874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:06.159190 22798 log.cc:824] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:06.159238 22723 log.cc:824] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:06.160017 22874 log.cc:824] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:06.161369 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: No bootstrap required, opened a new log
I20260620 08:59:06.161369 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: No bootstrap required, opened a new log
I20260620 08:59:06.161520 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: No bootstrap required, opened a new log
I20260620 08:59:06.165365 22723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.165365 22798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.165371 22874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.165578 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.146780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.98.61:46223)
0620 08:59:06.146882 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.146886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.148145 (+ 1259us) server_negotiation.cc:407] Connection header received
0620 08:59:06.148289 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.148291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.148335 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.148408 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.149121 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.149835 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.150922 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.151228 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.151298 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.154175 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.154179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.154182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.154206 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.155732 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.164079 (+ 8347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.164084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.164086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.164204 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.164519 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.164522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.164524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.164757 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.164938 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.165303 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.165432 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:59:06.165601 22723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f394d0bc2b041c8ba5f50b440c7f07d, State: Initialized, Role: FOLLOWER
I20260620 08:59:06.165611 22874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fe4b62ca24aa4be6a1212392873a9, State: Initialized, Role: FOLLOWER
I20260620 08:59:06.165601 22798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7f896ba9c44deab0a56d355d73f3bb, State: Initialized, Role: FOLLOWER
I20260620 08:59:06.165932 22874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.165932 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.165932 22723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.166613 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.166823 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.167101 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.167258 22874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:06.167433 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.167706 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.167582 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.167259 22723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:06.168687 22798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:06.169781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:59:06.172194 22696 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58920
I20260620 08:59:06.172405 22696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 08:59:06.172513 22696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 0 FOLLOWER]: Advancing to term 1
W20260620 08:59:06.173136 22676 tablet.cc:2369] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:59:06.174398 22696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
W20260620 08:59:06.174731 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:06.174839 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 08:59:06.175076 22919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:06.175144 22919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:59:06.175336 22696 leader_election.cc:288] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [CANDIDATE]: Term 1 election: Requested vote from peers 0f7f896ba9c44deab0a56d355d73f3bb (127.0.98.61:46223), 4f5fe4b62ca24aa4be6a1212392873a9 (127.0.98.60:36101)
I20260620 08:59:06.176131 22848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f5fe4b62ca24aa4be6a1212392873a9"
I20260620 08:59:06.176349 22848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:06.177646 22848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f394d0bc2b041c8ba5f50b440c7f07d in term 1.
I20260620 08:59:06.178125 22673 leader_election.cc:302] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f394d0bc2b041c8ba5f50b440c7f07d, 4f5fe4b62ca24aa4be6a1212392873a9; no voters:
I20260620 08:59:06.178483 22884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:06.178692 22884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [term 1 LEADER]: Becoming Leader. State: Replica: 1f394d0bc2b041c8ba5f50b440c7f07d, State: Running, Role: LEADER
I20260620 08:59:06.179028 22884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } }
I20260620 08:59:06.180814 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f394d0bc2b041c8ba5f50b440c7f07d. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.180930 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:06.181995 22772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0f7f896ba9c44deab0a56d355d73f3bb"
I20260620 08:59:06.182268 22772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:06.183634 22772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f394d0bc2b041c8ba5f50b440c7f07d in term 1.
I20260620 08:59:06.187683 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.177069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.0.98.62:39715)
0620 08:59:06.177180 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.177185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.178588 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 08:59:06.178786 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.178789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.178837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.178884 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.179910 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.180674 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.181846 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.182195 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.182272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.184202 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.184206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.184209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.184231 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.185844 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.186318 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.186322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.186324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.186404 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.186713 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.186717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.186718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.186932 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.187030 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.187376 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.187496 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:59:06.188082 22805 master_runner.cc:263] No leader master found from master 4f5fe4b62ca24aa4be6a1212392873a9
I20260620 08:59:06.188175 22805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:59:06.190121 22913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:06.190203 22913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:59:06.190490 22655 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260620 08:59:06.190553 22931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:06.190611 22655 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:59:06.191504 22931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:06.191517 22730 master_runner.cc:263] No leader master found from master 0f7f896ba9c44deab0a56d355d73f3bb
I20260620 08:59:06.191573 22730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:59:06.197674 22772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [term 1 FOLLOWER]: Refusing update from remote peer 1f394d0bc2b041c8ba5f50b440c7f07d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:06.198302 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.177648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.98.61:46223)
0620 08:59:06.177759 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.177763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.189102 (+ 11339us) server_negotiation.cc:407] Connection header received
0620 08:59:06.189262 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.189265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.189312 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.189357 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.190387 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.191064 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.192130 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.192481 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.192583 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.194068 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.194072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.194076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.194102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.195681 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.196140 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.196144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.196146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.196240 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.196975 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.196979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.196981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.197207 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.197321 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.198022 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.198135 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:59:06.198318 22848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [term 1 FOLLOWER]: Refusing update from remote peer 1f394d0bc2b041c8ba5f50b440c7f07d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:06.199003 22884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:06.201193 22884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:06.207139 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:06.184688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.0.98.60:36101)
0620 08:59:06.184912 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.184916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.197927 (+ 13011us) server_negotiation.cc:407] Connection header received
0620 08:59:06.198919 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.198922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.198968 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.199022 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.199843 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.200654 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.201713 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:06.202078 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:06.202145 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:06.203806 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:06.203810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:06.203813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:06.203839 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:06.205415 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.205866 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.205870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.205871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.205948 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:06.206228 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:06.206231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:06.206233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:06.206433 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:06.206539 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:06.206865 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 08:59:06.206984 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 08:59:06.207584 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f394d0bc2b041c8ba5f50b440c7f07d. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.207764 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.209214 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f394d0bc2b041c8ba5f50b440c7f07d. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.209340 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.211591 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.211750 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:06.213013 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.213128 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.213617 22936 mvcc.cc:205] Tried to move back new op lower bound from 7298850595617005568 to 7298850595552813056. Current Snapshot: MvccSnapshot[applied={T|T < 7298850595617005568}]
I20260620 08:59:06.215767 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.215873 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:06.216215 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f394d0bc2b041c8ba5f50b440c7f07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39715 } } peers { permanent_uuid: "0f7f896ba9c44deab0a56d355d73f3bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46223 } } peers { permanent_uuid: "4f5fe4b62ca24aa4be6a1212392873a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36101 } } }
I20260620 08:59:06.216324 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:06.218047 22934 mvcc.cc:205] Tried to move back new op lower bound from 7298850595617005568 to 7298850595552813056. Current Snapshot: MvccSnapshot[applied={T|T < 7298850595617005568}]
I20260620 08:59:06.222350 22931 catalog_manager.cc:1347] Generated new cluster ID: a5831fd9c1434952a079a6addee404f9
I20260620 08:59:06.222424 22931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:06.274575 22931 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:06.276520 22931 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:06.292914 22931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f394d0bc2b041c8ba5f50b440c7f07d: Generated new TSK 0
I20260620 08:59:06.293824 22931 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:59:06.325348 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.322140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.0.98.62:39715)
0620 08:59:06.322267 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.322271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.324069 (+ 1798us) server_negotiation.cc:407] Connection header received
0620 08:59:06.324469 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.324473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.324527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.324582 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.325225 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58936: BlockingRecv error: recv got EOF from 127.0.0.1:58936 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 08:59:06.326460 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.322859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.0.98.61:46223)
0620 08:59:06.322972 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.322976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.324386 (+ 1410us) server_negotiation.cc:407] Connection header received
0620 08:59:06.325730 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.325734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.325782 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.325828 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.326350 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52390: BlockingRecv error: recv got EOF from 127.0.0.1:52390 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 08:59:06.327762 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.323467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.98.60:36101)
0620 08:59:06.323623 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.323628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.326921 (+ 3293us) server_negotiation.cc:407] Connection header received
0620 08:59:06.327023 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.327026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.327076 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.327179 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.327654 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52774: BlockingRecv error: recv got EOF from 127.0.0.1:52774 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 08:59:06.332072 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.330556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.0.98.61:46223)
0620 08:59:06.330661 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.330665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.331117 (+ 452us) server_negotiation.cc:407] Connection header received
0620 08:59:06.331321 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.331325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.331404 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.331459 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.331961 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52392: BlockingRecv error: recv got EOF from 127.0.0.1:52392 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 08:59:06.337622 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.335808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.0.98.61:46223)
0620 08:59:06.335906 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.335910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.336451 (+ 541us) server_negotiation.cc:407] Connection header received
0620 08:59:06.336663 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.336667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.336724 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.336775 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.337521 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52400: BlockingRecv error: recv got EOF from 127.0.0.1:52400 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:59:06.343163 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.341639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.61:46223)
0620 08:59:06.341743 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.341748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.342260 (+ 512us) server_negotiation.cc:407] Connection header received
0620 08:59:06.342440 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.342444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.342495 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.342545 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.343044 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52402: BlockingRecv error: recv got EOF from 127.0.0.1:52402 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:59:06.351328 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.349481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.0.98.61:46223)
0620 08:59:06.349626 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.349631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.350204 (+ 573us) server_negotiation.cc:407] Connection header received
0620 08:59:06.350416 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.350420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.350477 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.350532 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.351216 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52410: BlockingRecv error: recv got EOF from 127.0.0.1:52410 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 08:59:06.363368 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.361726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.0.98.61:46223)
0620 08:59:06.361851 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.361856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.362403 (+ 547us) server_negotiation.cc:407] Connection header received
0620 08:59:06.362641 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.362644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.362695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.362743 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.363252 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52414: BlockingRecv error: recv got EOF from 127.0.0.1:52414 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:59:06.383437 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.381840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.0.98.61:46223)
0620 08:59:06.381958 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.381962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.382534 (+ 572us) server_negotiation.cc:407] Connection header received
0620 08:59:06.382652 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.382656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.382723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.382821 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.383317 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52422: BlockingRecv error: recv got EOF from 127.0.0.1:52422 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 08:59:06.418851 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.417665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.98.61:46223)
0620 08:59:06.417965 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.417969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.417987 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:59:06.418135 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.418139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.418189 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.418258 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.418731 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52434: BlockingRecv error: recv got EOF from 127.0.0.1:52434 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 08:59:06.486335 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.485158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.0.98.61:46223)
0620 08:59:06.485423 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.485429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.485454 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:59:06.485521 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.485525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.485590 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.485704 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.486215 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52444: BlockingRecv error: recv got EOF from 127.0.0.1:52444 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 08:59:06.617369 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.616343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.98.61:46223)
0620 08:59:06.616478 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.616482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.616624 (+ 142us) server_negotiation.cc:407] Connection header received
0620 08:59:06.616747 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.616750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.616795 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.616875 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.617274 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52458: BlockingRecv error: recv got EOF from 127.0.0.1:52458 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 08:59:06.876539 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:06.875528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.0.98.61:46223)
0620 08:59:06.875687 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:06.875691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:06.875758 (+ 67us) server_negotiation.cc:407] Connection header received
0620 08:59:06.875880 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:06.875882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:06.875928 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:06.875995 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:06.876426 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52464: BlockingRecv error: recv got EOF from 127.0.0.1:52464 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:59:07.176155 22907 catalog_manager.cc:1259] Loaded cluster ID: a5831fd9c1434952a079a6addee404f9
I20260620 08:59:07.176237 22907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: loading cluster ID for follower catalog manager: success
I20260620 08:59:07.176275 22919 catalog_manager.cc:1259] Loaded cluster ID: a5831fd9c1434952a079a6addee404f9
I20260620 08:59:07.176348 22919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: loading cluster ID for follower catalog manager: success
I20260620 08:59:07.177851 22907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: acquiring CA information for follower catalog manager: success
I20260620 08:59:07.177873 22919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: acquiring CA information for follower catalog manager: success
I20260620 08:59:07.178923 22907 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f5fe4b62ca24aa4be6a1212392873a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:59:07.178923 22919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f7f896ba9c44deab0a56d355d73f3bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:59:07.394210 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:07.390447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.0.98.61:46223)
0620 08:59:07.390688 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:07.390691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:07.390722 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:59:07.390843 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:07.390845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:07.390964 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:07.391064 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:07.391310 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.391865 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.392750 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.392990 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.393077 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:07.393386 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:07.393825 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:07.393917 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:07.394034 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:59:07.394090 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260620 08:59:07.400620 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:07.397219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.0.98.60:36101)
0620 08:59:07.397411 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:07.397414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:07.397426 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:59:07.397500 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:07.397502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:07.397611 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:07.397708 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:07.397855 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.398348 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.399205 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.399444 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.399537 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:07.399806 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:07.400242 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:07.400332 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:07.400426 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 08:59:07.400477 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260620 08:59:07.407977 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:07.404088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.0.98.62:39715)
0620 08:59:07.404282 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:07.404286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:07.404300 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:59:07.404471 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:07.404475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:07.404636 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:07.404744 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:07.405066 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.405644 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.406604 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.406827 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.406909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:07.407206 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:07.407556 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:07.407666 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:07.407758 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 08:59:07.407820 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260620 08:59:07.408290 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:07.404398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.0.98.61:46223)
0620 08:59:07.404521 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:07.404526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:07.404667 (+ 141us) server_negotiation.cc:407] Connection header received
0620 08:59:07.404840 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:07.404843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:07.404895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:07.404941 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:07.405255 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.405813 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.407048 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.407342 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.407411 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:07.407826 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:07.407901 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:07.407942 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:07.408117 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:59:07.408179 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:59:07.408905 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:07.404711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.98.60:36101)
0620 08:59:07.404813 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:07.404817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:07.405006 (+ 189us) server_negotiation.cc:407] Connection header received
0620 08:59:07.405114 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:07.405117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:07.405174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:07.405247 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:07.406226 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.406974 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.407821 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:07.408098 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:07.408170 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:07.408522 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:07.408593 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:07.408666 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:07.408743 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 08:59:07.408804 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:59:07.411090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22655
I20260620 08:59:07.416755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22730
I20260620 08:59:07.421981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 22805
2026-06-20T08:59:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3011 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:07 dist-test-slave-fb1v krb5kdc[23024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:07 dist-test-slave-fb1v krb5kdc[23024](info): set up 2 sockets
Jun 20 08:59:07 dist-test-slave-fb1v krb5kdc[23024](info): commencing operation
krb5kdc: starting...
W20260620 08:59:08.383987 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:08.503217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46853,127.0.98.61:35417,127.0.98.60:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:08.611845 23037 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:59:08.612159 23037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:08.612183 23037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:08.612223 23037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:08.617607 23037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:08.617702 23037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:08.617725 23037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:08.617745 23037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:08.617762 23037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:08.624418 23037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46853,127.0.98.61:35417,127.0.98.60:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:08.625487 23037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:08.626466 23037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:08.634742 23043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:08.634747 23042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:08.635075 23037 server_base.cc:1034] running on GCE node
W20260620 08:59:08.634742 23045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:08.635872 23037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:08.636950 23037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:08.638238 23037 hybrid_clock.cc:648] HybridClock initialized: now 1781945948638171 us; error 70 us; skew 500 ppm
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:08.641798 23037 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:08.642796 23037 webserver.cc:460] Webserver started at http://127.0.98.62:42759/ using document root <none> and password file <none>
I20260620 08:59:08.643365 23037 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:08.643435 23037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:08.643678 23037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:08.645737 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data/instance:
uuid: "478d079838504f2d9c150c0feb31e31d"
format_stamp: "Formatted at 2026-06-20 08:59:08 on dist-test-slave-fb1v"
server_key: "3d49215ce3825e958481072e5903d943"
server_key_iv: "8bf11d98f262bbf65f482625215afe11"
server_key_version: "encryptionkey@0"
I20260620 08:59:08.646251 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal/instance:
uuid: "478d079838504f2d9c150c0feb31e31d"
format_stamp: "Formatted at 2026-06-20 08:59:08 on dist-test-slave-fb1v"
server_key: "3d49215ce3825e958481072e5903d943"
server_key_iv: "8bf11d98f262bbf65f482625215afe11"
server_key_version: "encryptionkey@0"
I20260620 08:59:08.649901 23037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 08:59:08.652189 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:08.653309 23037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:59:08.653429 23037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal
uuid: "478d079838504f2d9c150c0feb31e31d"
format_stamp: "Formatted at 2026-06-20 08:59:08 on dist-test-slave-fb1v"
server_key: "3d49215ce3825e958481072e5903d943"
server_key_iv: "8bf11d98f262bbf65f482625215afe11"
server_key_version: "encryptionkey@0"
I20260620 08:59:08.653522 23037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:08.674163 23037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:08.678087 23037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:08.678295 23037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:08.686905 23037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46853
I20260620 08:59:08.687184 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46853 every 8 connection(s)
I20260620 08:59:08.687989 23037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:08.689334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 23037
I20260620 08:59:08.689421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:08.689688 392 external_mini_cluster.cc:1428] Setting key 17630b76c9a874bfaeab2d047329f369
I20260620 08:59:08.691695 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:08.697762 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:08.714578 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.700574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.0.98.62:46853)
0620 08:59:08.701015 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:08.701032 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:08.701206 (+ 174us) server_negotiation.cc:407] Connection header received
0620 08:59:08.702236 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:08.702242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:08.702450 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:08.702664 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:08.703532 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.704585 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.706442 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.706931 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.707067 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.709859 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:08.709867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:08.709882 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:08.709907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:08.711679 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.712114 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.712119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.712120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.712255 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.712585 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.712587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.712588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.713029 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:08.713144 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:08.713546 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 08:59:08.713740 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260620 08:59:08.714586 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.700574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46853 (local address 127.0.0.1:47960)
0620 08:59:08.701067 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.701104 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:08.701944 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:08.702718 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:08.702722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:08.703043 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:08.703385 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.703399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.704734 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.704737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.706095 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.706123 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.707117 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.707122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.707257 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.707884 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:08.707907 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:08.709692 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:08.711780 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.711788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.711796 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.712009 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.712373 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.712375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.712377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.712500 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.713122 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:08.713127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:08.713262 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 08:59:08.713426 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 08:59:08.713860 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":195,"spinlock_wait_cycles":65408,"thread_start_us":160,"threads_started":1}
I20260620 08:59:08.717486 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 08:59:08.718616 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.718189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50162)
0620 08:59:08.718367 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.718493 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 08:59:08.718990 23056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
I20260620 08:59:08.719396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35417
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46853,127.0.98.61:35417,127.0.98.60:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:59:08.720321 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.720007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50164)
0620 08:59:08.720174 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.720236 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 08:59:08.720484 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35417: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
I20260620 08:59:08.769773 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } attempt: 1
I20260620 08:59:08.770617 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.770322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50178)
0620 08:59:08.770474 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.770540 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 08:59:08.771418 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.771201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50194)
0620 08:59:08.771314 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.771362 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 08:59:08.771557 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35417: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
W20260620 08:59:08.831447 23112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:59:08.831770 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:08.831797 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:08.831856 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:08.835870 23112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:08.835919 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:08.835937 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:08.835951 23112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:08.835985 23112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:08.839869 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } attempt: 2
I20260620 08:59:08.840919 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.840567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50206)
0620 08:59:08.840770 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.840838 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 08:59:08.841408 23112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46853,127.0.98.61:35417,127.0.98.60:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:08.841634 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.841400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50220)
0620 08:59:08.841509 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.841561 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 08:59:08.841751 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35417: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35417: connect: Connection refused (error 111)
I20260620 08:59:08.842495 23112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:08.843441 23112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:08.851907 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:08.851902 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:08.851902 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:08.852203 23112 server_base.cc:1034] running on GCE node
I20260620 08:59:08.852718 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:08.853770 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:08.855024 23112 hybrid_clock.cc:648] HybridClock initialized: now 1781945948854979 us; error 35 us; skew 500 ppm
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:08.858013 23112 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 08:59:08.858855 23112 webserver.cc:460] Webserver started at http://127.0.98.61:35701/ using document root <none> and password file <none>
I20260620 08:59:08.859375 23112 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:08.859439 23112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:08.859649 23112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:08.861596 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data/instance:
uuid: "d4e35ed6120d4e9b8f00284f324ad21e"
format_stamp: "Formatted at 2026-06-20 08:59:08 on dist-test-slave-fb1v"
server_key: "1d82e6f129949b11bda5b577d028d1b8"
server_key_iv: "be0354eba3aad7fcf3e4bcd9084edc12"
server_key_version: "encryptionkey@0"
I20260620 08:59:08.862123 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal/instance:
uuid: "d4e35ed6120d4e9b8f00284f324ad21e"
format_stamp: "Formatted at 2026-06-20 08:59:08 on dist-test-slave-fb1v"
server_key: "1d82e6f129949b11bda5b577d028d1b8"
server_key_iv: "be0354eba3aad7fcf3e4bcd9084edc12"
server_key_version: "encryptionkey@0"
I20260620 08:59:08.865638 23112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 08:59:08.867982 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:08.869174 23112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:59:08.869282 23112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal
uuid: "d4e35ed6120d4e9b8f00284f324ad21e"
format_stamp: "Formatted at 2026-06-20 08:59:08 on dist-test-slave-fb1v"
server_key: "1d82e6f129949b11bda5b577d028d1b8"
server_key_iv: "be0354eba3aad7fcf3e4bcd9084edc12"
server_key_version: "encryptionkey@0"
I20260620 08:59:08.869372 23112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:08.882036 23112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:08.884927 23112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:08.885076 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:08.892390 23112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35417
I20260620 08:59:08.892660 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35417 every 8 connection(s)
I20260620 08:59:08.893293 23112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/data/info.pb
I20260620 08:59:08.895744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 23112
I20260620 08:59:08.895838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-1/wal/instance
I20260620 08:59:08.896112 392 external_mini_cluster.cc:1428] Setting key 37a8ccdb03beb13b978f9f5dfa02fb92
I20260620 08:59:08.896113 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:08.900741 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:08.914427 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.903068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.0.98.62:46853)
0620 08:59:08.903235 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:08.903240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:08.904055 (+ 815us) server_negotiation.cc:407] Connection header received
0620 08:59:08.905081 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:08.905085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:08.905139 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:08.905214 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:08.906374 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.906947 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.908205 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.908441 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.908533 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.911108 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:08.911111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:08.911113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:08.911138 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:08.912493 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.912966 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.912969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.912971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.913057 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.913358 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.913361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.913363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.913541 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:08.913623 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:08.914132 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 08:59:08.914260 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:59:08.915757 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.903370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46853 (local address 127.0.0.1:47974)
0620 08:59:08.903892 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.903951 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:08.904891 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:08.905377 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:08.905384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:08.905708 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:08.906238 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.906251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.907064 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.907067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.908059 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.908071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.908589 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.908594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.908790 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.909376 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:08.909395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:08.910938 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:08.912602 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.912608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.912623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.912877 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.913139 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.913142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.913144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.913276 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.913635 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:08.913641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:08.913894 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 08:59:08.914066 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 08:59:08.914387 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":121,"threads_started":1}
I20260620 08:59:08.917913 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 08:59:08.924794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9wbfbl/build/debug/bin/kudu
/tmp/dist-test-task9wbfbl/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44697
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46853,127.0.98.61:35417,127.0.98.60:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:59:08.928071 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.918686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50222)
0620 08:59:08.918832 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.918845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:08.918910 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:08.919435 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:08.919438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:08.919599 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:08.919740 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.919746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.920825 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.920828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.921359 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.921366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.921921 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.921925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.922118 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.922551 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:08.922562 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:08.924128 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:08.926578 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.926581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.926583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.926797 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.927073 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.927076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.927078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.927148 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.927554 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:08.927557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:08.927669 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 08:59:08.927762 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:59:08.927893 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":46}
I20260620 08:59:08.928160 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.918807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.0.98.61:35417)
0620 08:59:08.919122 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:08.919130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:08.919146 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:59:08.919175 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:08.919179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:08.919323 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:08.919436 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:08.919837 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.920700 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.921460 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.921822 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.921912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.924347 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:08.924358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:08.924373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:08.924398 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:08.926466 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.926900 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.926905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.926907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.926997 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.927220 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.927223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.927225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.927460 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:08.927579 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:08.927847 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 08:59:08.928000 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":239,"thread_start_us":147,"threads_started":1}
I20260620 08:59:08.930466 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:08.931191 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.930938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53744)
0620 08:59:08.931053 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.931128 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 08:59:08.931375 23131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
I20260620 08:59:08.932250 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.932051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53760)
0620 08:59:08.932148 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.932197 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:59:08.932377 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
I20260620 08:59:08.932917 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } attempt: 3
Jun 20 08:59:08 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:59:08.940508 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.933473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.0.98.61:35417)
0620 08:59:08.933574 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:08.933578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:08.933590 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:59:08.933723 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:08.933727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:08.933773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:08.933836 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:08.934229 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.934736 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.935466 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.935764 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.935848 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.937640 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:08.937644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:08.937646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:08.937669 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:08.939078 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.939448 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.939452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.939454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.939522 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:08.939784 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:08.939787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:08.939788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:08.939953 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:08.940033 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:08.940279 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 08:59:08.940373 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:59:08.940510 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.933387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50228)
0620 08:59:08.933520 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.933534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:08.933630 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:08.933861 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:08.933863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:08.934029 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:08.934128 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.934133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.934831 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.934835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.935367 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:08.935374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:08.935867 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:08.935871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:08.935969 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:08.936403 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:08.936412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:08.937525 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:08.939187 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.939191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.939193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.939367 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.939598 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:08.939601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:08.939630 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:08.939708 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:08.940033 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:08.940037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:08.940128 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 08:59:08.940221 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:59:08.940351 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 08:59:08.941917 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:08.942531 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.942336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53762)
0620 08:59:08.942432 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.942476 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 08:59:08.943120 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.942910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53772)
0620 08:59:08.943028 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.943071 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 08:59:08.943219 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
I20260620 08:59:08.974380 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } attempt: 1
I20260620 08:59:08.975247 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.974934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53778)
0620 08:59:08.975109 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.975171 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 08:59:08.975935 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.975744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53780)
0620 08:59:08.975843 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.975884 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:59:08.976048 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
I20260620 08:59:08.981737 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } attempt: 1
I20260620 08:59:08.982810 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.982427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53786)
0620 08:59:08.982589 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.982652 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 08:59:08.983937 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:08.983487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53800)
0620 08:59:08.983698 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:08.983759 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 08:59:08.984117 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
W20260620 08:59:09.038007 23186 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:59:09.038306 23186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:09.038368 23186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:09.038434 23186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:09.042630 23186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:09.042695 23186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:09.042721 23186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:09.042743 23186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:09.042773 23186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:09.048437 23186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46853,127.0.98.61:35417,127.0.98.60:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:09.049471 23186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:09.050416 23186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:59:09.051215 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } attempt: 2
I20260620 08:59:09.052038 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.051740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53812)
0620 08:59:09.051897 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.051964 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 08:59:09.052284 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } attempt: 2
I20260620 08:59:09.052986 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.052707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53814)
0620 08:59:09.052845 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.052905 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 08:59:09.053113 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.052896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53824)
0620 08:59:09.053003 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.053051 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 08:59:09.053220 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
I20260620 08:59:09.053717 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.053488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53830)
0620 08:59:09.053597 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.053657 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 08:59:09.053969 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44697: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44697: connect: Connection refused (error 111)
W20260620 08:59:09.059672 23193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:09.059636 23195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:09.059650 23192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:09.059847 23186 server_base.cc:1034] running on GCE node
I20260620 08:59:09.060544 23186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:09.061599 23186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:09.062786 23186 hybrid_clock.cc:648] HybridClock initialized: now 1781945949062777 us; error 48 us; skew 500 ppm
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:09.065824 23186 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 08:59:09.067126 23186 webserver.cc:460] Webserver started at http://127.0.98.60:38755/ using document root <none> and password file <none>
I20260620 08:59:09.067924 23186 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:09.068003 23186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:09.068243 23186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:09.070853 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data/instance:
uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995"
format_stamp: "Formatted at 2026-06-20 08:59:09 on dist-test-slave-fb1v"
server_key: "afa7626b6783159b81c8fc78abf12bd6"
server_key_iv: "768980914d5de91f84180670cce03f40"
server_key_version: "encryptionkey@0"
I20260620 08:59:09.071779 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal/instance:
uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995"
format_stamp: "Formatted at 2026-06-20 08:59:09 on dist-test-slave-fb1v"
server_key: "afa7626b6783159b81c8fc78abf12bd6"
server_key_iv: "768980914d5de91f84180670cce03f40"
server_key_version: "encryptionkey@0"
I20260620 08:59:09.076169 23186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 08:59:09.078701 23202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:09.080618 23186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:59:09.080763 23186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data,/tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal
uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995"
format_stamp: "Formatted at 2026-06-20 08:59:09 on dist-test-slave-fb1v"
server_key: "afa7626b6783159b81c8fc78abf12bd6"
server_key_iv: "768980914d5de91f84180670cce03f40"
server_key_version: "encryptionkey@0"
I20260620 08:59:09.080907 23186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:09.098832 23186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:09.102763 23186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:09.102955 23186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:09.110229 23186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44697
I20260620 08:59:09.110504 23254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44697 every 8 connection(s)
I20260620 08:59:09.111220 23186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/data/info.pb
I20260620 08:59:09.112191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9wbfbl/build/debug/bin/kudu as pid 23186
I20260620 08:59:09.112356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9wbfbl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945671001899-392-0/minicluster-data/master-2/wal/instance
I20260620 08:59:09.112740 392 external_mini_cluster.cc:1428] Setting key 858d48414da93fb1abe2d65281db01fc
I20260620 08:59:09.115382 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:09.119344 23255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:09.129122 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.114406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.0.98.60:44697)
0620 08:59:09.115003 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.115029 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.115086 (+ 57us) server_negotiation.cc:407] Connection header received
0620 08:59:09.116068 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.116094 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.116593 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.117015 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.118166 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.119252 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.120279 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.120762 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.120932 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.123767 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.123774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.123787 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.123826 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.125791 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.126214 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.126221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.126227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.126357 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.126766 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.126771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.126773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.127474 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.127677 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.127982 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.128293 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":149,"threads_started":1}
I20260620 08:59:09.130478 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.121071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.0.98.62:46853)
0620 08:59:09.121216 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.121221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.121554 (+ 333us) server_negotiation.cc:407] Connection header received
0620 08:59:09.121720 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.121723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.121783 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.121883 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.123047 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.123808 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.124976 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.125205 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.125289 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.127437 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.127439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.127442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.127465 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.128819 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.129245 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.129249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.129251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.129347 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.129625 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.129636 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.129639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.129835 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.129921 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.130275 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.130368 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:59:09.130539 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.121023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46853 (local address 127.0.0.1:47988)
0620 08:59:09.121434 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.121482 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:09.121610 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:09.122078 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:09.122085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:09.122591 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:09.122776 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.122784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.123942 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.123945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.124835 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.124845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.125317 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.125322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.125431 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.125863 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:09.125878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:09.127302 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:09.128914 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.128922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.128927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.129142 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.129454 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.129458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.129459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.129536 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.129934 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:09.129940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:09.130055 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 08:59:09.130204 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 08:59:09.130368 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":138,"threads_started":1}
I20260620 08:59:09.131587 23255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:59:09.140077 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.132061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35417 (local address 127.0.0.1:50230)
0620 08:59:09.132233 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.132245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:09.132310 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:09.132527 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:09.132528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:09.132653 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:09.132744 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.132749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.133509 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.133513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.134190 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.134198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.134679 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.134683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.134777 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.135121 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:09.135134 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:09.136664 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:09.138371 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.138375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.138376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.138605 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.138991 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.138994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.138996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.139111 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.139528 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:09.139531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:09.139645 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 08:59:09.139798 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 08:59:09.139930 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 08:59:09.140152 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.132157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.98.61:35417)
0620 08:59:09.132263 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.132267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.132278 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:59:09.132399 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.132402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.132448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.132520 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.132837 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.133415 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.134304 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.134558 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.134648 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.136775 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.136780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.136783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.136809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.138260 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.138802 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.138806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.138808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.138891 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.139198 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.139202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.139203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.139425 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.139517 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.139863 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.139985 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:59:09.140362 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.131389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.0.98.62:46853)
0620 08:59:09.131489 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.131492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.131510 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:59:09.131633 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.131635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.131679 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.131753 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.132428 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.133057 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.133962 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.134191 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.134297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.136497 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.136501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.136503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.136525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.138092 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.138643 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.138649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.138651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.138821 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.139136 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.139139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.139140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.139367 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.139600 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.140115 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.140229 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:59:09.140769 23255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:59:09.145112 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } attempt: 3
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 08:59:09.149201 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.141365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.0.98.60:44697)
0620 08:59:09.141451 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.141456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.141467 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:59:09.141492 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.141507 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.141541 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.141598 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.142040 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.142983 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.144006 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.144370 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.144468 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.146405 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.146408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.146410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.146429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.147677 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.148060 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.148063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.148065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.148137 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.148392 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.148395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.148396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.148591 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.148693 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.148922 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.149041 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":36}
I20260620 08:59:09.149325 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.141246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53846)
0620 08:59:09.141331 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.141344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:09.141411 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:09.141619 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:09.141622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:09.141756 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:09.141826 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.141833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.143090 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.143094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.143907 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.143916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.144468 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.144470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.144523 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.144895 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:09.144908 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:09.146284 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:09.147801 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.147804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.147806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.147981 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.148214 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.148217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.148219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.148311 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.148684 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:09.148687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:09.148768 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 08:59:09.148851 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:59:09.149208 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 08:59:09.151600 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.141750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.0.98.61:35417)
0620 08:59:09.141885 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.141889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.141901 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:59:09.141932 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.141935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.141971 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.142021 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.142941 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.143690 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.144632 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.144902 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.145007 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.148183 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.148187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.148189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.148212 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.150231 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.150623 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.150625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.150626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.150690 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.150932 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.150934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.150935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.151084 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.151232 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.151398 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.151481 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:59:09.152359 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } attempt: 3
I20260620 08:59:09.152674 23255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: Bootstrap starting.
I20260620 08:59:09.154783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:59:09.155364 23255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: Neither blocks nor log segments found. Creating new log.
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 08:59:09.156261 23255 log.cc:824] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:09.155843 23079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "478d079838504f2d9c150c0feb31e31d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002
Jun 20 08:59:09 dist-test-slave-fb1v krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 08:59:09.160110 23255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: No bootstrap required, opened a new log
I20260620 08:59:09.161019 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.152745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53856)
0620 08:59:09.152843 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.152857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:09.152928 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:09.153206 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:09.153209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:09.153387 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:09.153473 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.153478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.154213 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.154216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.154871 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.154881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.155371 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.155375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.155456 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.155895 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:09.155908 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:09.157065 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:09.159430 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.159434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.159436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.159717 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.160027 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.160030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.160032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.160132 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.160526 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:09.160529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:09.160627 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 08:59:09.160732 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:59:09.160868 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 08:59:09.161237 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.152909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.0.98.60:44697)
0620 08:59:09.153004 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.153007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.153018 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:59:09.153046 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.153048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.153089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.153558 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.153624 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.154123 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.154964 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.155261 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.155364 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.157216 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.157220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.157223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.157246 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.159042 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.159823 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.159827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.159829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.159915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.160223 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.160226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.160228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.160433 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.160529 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.160979 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.161096 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":768}
I20260620 08:59:09.161667 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.145455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44697 (local address 127.0.0.1:53850)
0620 08:59:09.145577 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:09.145590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:09.145661 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:09.146569 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:09.146573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:09.146740 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:09.146839 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.146845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.148358 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.148361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.149228 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:09.149239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.149796 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.149805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:09.149892 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.156368 (+ 6476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:09.156383 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:09.157957 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:09.160270 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.160274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.160276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.160493 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.160757 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:09.160760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:09.160762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:09.160838 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:09.161206 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:09.161208 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:09.161305 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 08:59:09.161384 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:59:09.161516 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 08:59:09.162108 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.145652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.0.98.60:44697)
0620 08:59:09.145965 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.145974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.145987 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:59:09.146394 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.146397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.146443 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.146940 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.147027 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.148260 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.149417 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.149676 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.149772 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.158078 (+ 8306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.158082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.158085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.158109 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.160160 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.160612 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.160615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.160616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.160679 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.160917 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.160920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.160921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.161098 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.161826 (+ 728us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.161894 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.161982 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":155,"threads_started":1}
I20260620 08:59:09.164177 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d: Bootstrap starting.
I20260620 08:59:09.164256 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: Bootstrap starting.
I20260620 08:59:09.164357 23255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.164587 23255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fbf2cd1d4a40d8bbe145d23d0c8995, State: Initialized, Role: FOLLOWER
I20260620 08:59:09.164924 23255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.165411 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.165534 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.166263 23255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:09.166401 23180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:09.166507 23105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:09.167019 23180 log.cc:824] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:09.167302 23105 log.cc:824] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:09.169659 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: No bootstrap required, opened a new log
W20260620 08:59:09.171101 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:09.171185 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:59:09.169967 23186 master_runner.cc:186] Error getting master registration for 127.0.98.62:46853: OK, instance_id { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" instance_seqno: 1781945948684517 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:59:09.172813 23186 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:59:09.173787 23180 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.174013 23180 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e35ed6120d4e9b8f00284f324ad21e, State: Initialized, Role: FOLLOWER
I20260620 08:59:09.174371 23180 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.174914 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.175055 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.175807 23180 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:09.180680 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d: No bootstrap required, opened a new log
W20260620 08:59:09.181573 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:09.181648 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:59:09.179669 23112 master_runner.cc:186] Error getting master registration for 127.0.98.62:46853: OK, instance_id { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" instance_seqno: 1781945948684517 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:59:09.183360 23112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:59:09.184293 23105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.184640 23105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478d079838504f2d9c150c0feb31e31d, State: Initialized, Role: FOLLOWER
I20260620 08:59:09.185067 23105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.186411 23105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:09.186496 23079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 08:59:09.186610 23079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:09.187939 23079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.189893 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "478d079838504f2d9c150c0feb31e31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d4e35ed6120d4e9b8f00284f324ad21e"
I20260620 08:59:09.190129 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:09.191464 23154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 478d079838504f2d9c150c0feb31e31d in term 1.
I20260620 08:59:09.192021 23079 leader_election.cc:288] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [CANDIDATE]: Term 1 election: Requested vote from peers d4e35ed6120d4e9b8f00284f324ad21e (127.0.98.61:35417), 16fbf2cd1d4a40d8bbe145d23d0c8995 (127.0.98.60:44697)
I20260620 08:59:09.192282 23056 leader_election.cc:302] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478d079838504f2d9c150c0feb31e31d, d4e35ed6120d4e9b8f00284f324ad21e; no voters:
I20260620 08:59:09.192467 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.192586 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.192662 23290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:09.192727 23290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [term 1 LEADER]: Becoming Leader. State: Replica: 478d079838504f2d9c150c0feb31e31d, State: Running, Role: LEADER
I20260620 08:59:09.193029 23290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } }
I20260620 08:59:09.193194 23229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "478d079838504f2d9c150c0feb31e31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995"
I20260620 08:59:09.193434 23229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:09.194697 23229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 478d079838504f2d9c150c0feb31e31d in term 1.
I20260620 08:59:09.198055 23037 master_runner.cc:263] No leader master found from master 478d079838504f2d9c150c0feb31e31d
I20260620 08:59:09.198122 23037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:59:09.199996 23301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:09.200073 23301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:59:09.201324 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478d079838504f2d9c150c0feb31e31d. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.201452 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:09.201946 23309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:09.203389 23309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:09.214388 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.204199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.0.98.62:46853)
0620 08:59:09.204420 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.204425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.205338 (+ 913us) server_negotiation.cc:407] Connection header received
0620 08:59:09.205546 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.205549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.205599 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.205664 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.206648 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.207420 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.208619 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.208955 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.209023 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.210824 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.210827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.210831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.210858 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.212454 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.213059 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.213063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.213065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.213148 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.213426 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.213442 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.213444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.213652 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.213749 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.214089 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.214220 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:59:09.223569 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.205404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.0.98.60:44697)
0620 08:59:09.205519 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.205522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.215195 (+ 9673us) server_negotiation.cc:407] Connection header received
0620 08:59:09.215366 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.215369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.215425 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.215489 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.216235 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.217034 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.218123 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.218491 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.218564 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.220141 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.220145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.220148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.220175 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.221801 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.222262 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.222266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.222268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.222347 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.222632 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.222635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.222637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.222840 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.222947 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.223264 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.223390 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:59:09.227247 23229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [term 1 FOLLOWER]: Refusing update from remote peer 478d079838504f2d9c150c0feb31e31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:09.227689 23154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [term 1 FOLLOWER]: Refusing update from remote peer 478d079838504f2d9c150c0feb31e31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:09.228219 23303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:09.228672 23303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:09.231505 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:09.204860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.0.98.61:35417)
0620 08:59:09.204973 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.204977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.211879 (+ 6902us) server_negotiation.cc:407] Connection header received
0620 08:59:09.212050 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.212067 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.212118 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.212323 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.224272 (+ 11949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.225010 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.226044 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:09.226335 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:09.226410 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:09.228197 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:09.228201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:09.228204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:09.228228 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:09.229747 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.230205 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.230209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.230211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.230292 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:09.230582 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:09.230585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:09.230587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:09.230797 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:09.230911 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:09.231233 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 08:59:09.231349 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:59:09.232511 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478d079838504f2d9c150c0feb31e31d. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.232640 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.232686 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478d079838504f2d9c150c0feb31e31d. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.232785 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.235899 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.236021 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:09.236944 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.237061 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:09.237090 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.237192 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.237820 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "478d079838504f2d9c150c0feb31e31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478d079838504f2d9c150c0feb31e31d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } peers { permanent_uuid: "d4e35ed6120d4e9b8f00284f324ad21e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35417 } } peers { permanent_uuid: "16fbf2cd1d4a40d8bbe145d23d0c8995" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44697 } } }
I20260620 08:59:09.237926 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:59:09.239935 23314 mvcc.cc:205] Tried to move back new op lower bound from 7298850608027996160 to 7298850607922978816. Current Snapshot: MvccSnapshot[applied={T|T < 7298850608027996160}]
I20260620 08:59:09.247866 23309 catalog_manager.cc:1347] Generated new cluster ID: e745e954afbe49ea873252b7f9678600
I20260620 08:59:09.247942 23309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:09.249637 23315 mvcc.cc:205] Tried to move back new op lower bound from 7298850608027996160 to 7298850607922978816. Current Snapshot: MvccSnapshot[applied={T|T < 7298850608027996160}]
I20260620 08:59:09.271523 23309 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:09.273082 23309 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:09.288334 23309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 478d079838504f2d9c150c0feb31e31d: Generated new TSK 0
I20260620 08:59:09.289321 23309 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:59:09.322652 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.319492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.0.98.62:46853)
0620 08:59:09.319667 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.319672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.321520 (+ 1848us) server_negotiation.cc:407] Connection header received
0620 08:59:09.321747 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.321750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.321804 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.321856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.322496 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48024: BlockingRecv error: recv got EOF from 127.0.0.1:48024 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 08:59:09.324168 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.320247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.98.61:35417)
0620 08:59:09.320353 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.320357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.323024 (+ 2667us) server_negotiation.cc:407] Connection header received
0620 08:59:09.323192 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.323196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.323244 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.323292 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.324058 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50256: BlockingRecv error: recv got EOF from 127.0.0.1:50256 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 08:59:09.328619 23260 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.320846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.98.60:44697)
0620 08:59:09.320955 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.320959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.323840 (+ 2881us) server_negotiation.cc:407] Connection header received
0620 08:59:09.327870 (+ 4030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.327873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.327928 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.327977 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.328490 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53872: BlockingRecv error: recv got EOF from 127.0.0.1:53872 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 08:59:09.333024 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.331400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.0.98.61:35417)
0620 08:59:09.331509 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.331513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.332095 (+ 582us) server_negotiation.cc:407] Connection header received
0620 08:59:09.332302 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.332306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.332381 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.332429 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.332920 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50264: BlockingRecv error: recv got EOF from 127.0.0.1:50264 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 08:59:09.338423 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.336870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.0.98.61:35417)
0620 08:59:09.336974 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.336979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.337508 (+ 529us) server_negotiation.cc:407] Connection header received
0620 08:59:09.337705 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.337708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.337764 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.337818 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.338323 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50266: BlockingRecv error: recv got EOF from 127.0.0.1:50266 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 08:59:09.344324 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.342679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.0.98.61:35417)
0620 08:59:09.342791 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.342795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.343348 (+ 553us) server_negotiation.cc:407] Connection header received
0620 08:59:09.343534 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.343537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.343590 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.343697 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.344218 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50280: BlockingRecv error: recv got EOF from 127.0.0.1:50280 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 08:59:09.357208 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.351007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.0.98.61:35417)
0620 08:59:09.351219 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.351246 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.352816 (+ 1570us) server_negotiation.cc:407] Connection header received
0620 08:59:09.353114 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.353117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.353296 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.353383 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.354198 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50284: BlockingRecv error: recv got EOF from 127.0.0.1:50284 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 08:59:09.370057 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.368097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.0.98.61:35417)
0620 08:59:09.368254 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.368259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.369034 (+ 775us) server_negotiation.cc:407] Connection header received
0620 08:59:09.369243 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.369247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.369306 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.369359 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.369945 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50286: BlockingRecv error: recv got EOF from 127.0.0.1:50286 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 08:59:09.389755 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.388584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.0.98.61:35417)
0620 08:59:09.388709 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.388714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.388941 (+ 227us) server_negotiation.cc:407] Connection header received
0620 08:59:09.389080 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.389084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.389136 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.389204 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.389644 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50300: BlockingRecv error: recv got EOF from 127.0.0.1:50300 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 08:59:09.424762 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.423690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.0.98.61:35417)
0620 08:59:09.423879 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.423883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.423900 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:59:09.424041 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.424044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.424095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.424181 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.424646 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50312: BlockingRecv error: recv got EOF from 127.0.0.1:50312 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 08:59:09.491668 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.490607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.61:35417)
0620 08:59:09.490753 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.490758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.490851 (+ 93us) server_negotiation.cc:407] Connection header received
0620 08:59:09.490996 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.490999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.491052 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.491118 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.491543 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50320: BlockingRecv error: recv got EOF from 127.0.0.1:50320 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 08:59:09.622785 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.621686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.0.98.61:35417)
0620 08:59:09.621831 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.621835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.621962 (+ 127us) server_negotiation.cc:407] Connection header received
0620 08:59:09.622106 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.622110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.622158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.622228 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.622668 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50336: BlockingRecv error: recv got EOF from 127.0.0.1:50336 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 08:59:09.881584 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:59:09.880766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.0.98.61:35417)
0620 08:59:09.880887 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:09.880891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:09.880906 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:59:09.881000 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:09.881002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:09.881047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:09.881114 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:09.881481 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35598: BlockingRecv error: recv got EOF from 127.0.0.1:35598 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:59:10.172500 23275 catalog_manager.cc:1259] Loaded cluster ID: e745e954afbe49ea873252b7f9678600
I20260620 08:59:10.172580 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: loading cluster ID for follower catalog manager: success
I20260620 08:59:10.174311 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: acquiring CA information for follower catalog manager: success
I20260620 08:59:10.175418 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16fbf2cd1d4a40d8bbe145d23d0c8995: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:59:10.182819 23288 catalog_manager.cc:1259] Loaded cluster ID: e745e954afbe49ea873252b7f9678600
I20260620 08:59:10.182885 23288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: loading cluster ID for follower catalog manager: success
I20260620 08:59:10.184571 23288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: acquiring CA information for follower catalog manager: success
I20260620 08:59:10.185773 23288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4e35ed6120d4e9b8f00284f324ad21e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:59:10.399343 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:10.395598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.0.98.61:35417)
0620 08:59:10.395879 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:10.395884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:10.395899 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:59:10.396061 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:10.396063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:10.396188 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:10.396284 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:10.396543 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.397070 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.397908 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.398136 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.398222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:10.398516 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:10.398943 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:10.399062 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:10.399133 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:59:10.399192 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260620 08:59:10.405673 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:10.402158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.0.98.60:44697)
0620 08:59:10.402394 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:10.402398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:10.402410 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:59:10.402481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:10.402484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:10.402589 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:10.402678 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:10.402845 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.403341 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.404166 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.404391 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.404479 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:10.404826 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:10.405273 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:10.405382 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:10.405455 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 08:59:10.405507 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260620 08:59:10.412415 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:10.408695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.0.98.61:35417)
0620 08:59:10.408805 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:10.408809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:10.408878 (+ 69us) server_negotiation.cc:407] Connection header received
0620 08:59:10.409030 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:10.409033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:10.409090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:10.409137 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:10.409408 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.410108 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.411280 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.411502 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.411595 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:10.411939 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:10.412023 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:10.412087 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:10.412226 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 08:59:10.412281 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:59:10.412567 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:10.408428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.98.62:46853)
0620 08:59:10.408686 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:10.408690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:10.408705 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:59:10.408793 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:10.408796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:10.408943 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:10.409303 (+ 360us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:10.409408 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.410030 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.411235 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.411488 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.411573 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:10.411921 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:10.412236 (+ 315us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:10.412311 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:10.412379 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 08:59:10.412432 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260620 08:59:10.413013 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:10.409010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.0.98.60:44697)
0620 08:59:10.409124 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:10.409128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:10.409334 (+ 206us) server_negotiation.cc:407] Connection header received
0620 08:59:10.409491 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:10.409494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:10.409568 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:10.409641 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:10.409837 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.410439 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.411681 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:10.411976 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:10.412060 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:10.412263 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:10.412338 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:10.412396 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:10.412847 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 08:59:10.412902 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:59:10.421192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 23037
I20260620 08:59:10.426820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 23112
I20260620 08:59:10.431874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9wbfbl/build/debug/bin/kudu with pid 23186
2026-06-20T08:59:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3009 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6021 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279418 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST